Storage¶
Pantavisor needs some persistent artifacts on disk to perform several operations such as storing revisions, updates, disks, logs and other RW operations.
These artifacts are stored in the PVROOT
partition, which is mounted to /storage
by Pantavisor during initialization.
Its structure is as follows:
.
├── boot # bootloader files
│ ├── boot.scr
│ └── uboot.txt
├── cache # metadata
│ ├── devmeta
│ ├── dropbear
│ └── usrmeta
├── config # Pantacor Hub RW configuration
│ └── pantahub.config
├── disks # persistent container and Pantavisor storage
├── logs # container and Pantavisor logs are centralized here
├── objects # general revision artifacts
│ ├── 1dc4107c036a14aac38b5218fe50a8c5fa9e73d92a1d24337a57cd833c0a8797
│ ├── 2c0236e022c8bed4af695a96504ee7ece2b43e0d57f5a4d4181edb028539f061
│ ├── 8debde333d3885abf03cb7f954017744c8e2fc6708f89f0a4771b0bf367fd2a6
│ ├── 97bf54227c91cd93b540785942e4884ac6a7352fcea84afc3e0b8d174c7e4825
│ ├── ac1a09142f25a8df6742a3da290008f99c05c7b83d2754d9928f1a081714eaed
│ ├── b236948bd3caaf31c00af82fbeeb0816962e39a96ad6c5cf9bfc1a547cdbe005
│ ├── b6959af6ef61ae8ff729a9e5e41a8fdf5f318bd6ef1d21d38dc796d0db7e7627
│ └── f4b9d24c631cd1bcc303cedd3b9b3c6d32556b1975ac4262e823bf5d9affbc9f
├── trails # state JSON of each revision with objects hardlinks
│ └── 0 # Pantavisor checkout for revision 0
│ ├── _sigs
│ │ └── bsp.json # .json files are shipped inline, no hardlink
│ ├── bsp
│ │ ├── drivers.json
│ │ ├── firmware.squashfs # this is a hardlink to an object
│ │ ├── modules_6.1.77+.squashfs
│ │ ├── modules_6.1.77-v7+.squashfs
│ │ ├── modules_6.1.77-v7l+.squashfs
│ │ ├── modules_6.1.77-v8+.squashfs
│ │ ├── modules_6.1.77-v8-16k+.squashfs
│ │ ├── pantavisor-rpi.img.gz
│ │ ├── run.json
│ │ └── trail.config
│ ├── device.json
│ ├── .pvr/json # full revision state JSON
│ ├── <container-1-name>
│ ├── <container-2-name>
│ └── ...
...
Boot¶
Contains revision number information written by Pantavisor and consumed by the bootloader so it can run the proper revision after powering off and rebooting.
Cache¶
Mainly to store information that is not fixed to any given revision.
Metadata¶
Metadata is the information that is exchanged between Pantavisor and the outside world, either remotely or locally, that is not attached to a revision itself.
Metadata is just a list of key-value pairs of string type. This list is persistent and stored on disk. By default, it is saved in plain text, but this can be changed in favour of volumes on a crypt disk.
Device metadata¶
This type of metadata is meant to be generated by Pantavisor or any of the running containers to expose internal information.
The list of device metadata is persistent on disk and can be managed locally by mgmt containers. Their values will be uploaded to the cloud in remote mode.
Check the list of default key-value pairs that Pantavisor sends.
User metadata¶
This type of metadata is meant to be generated by the user so Pantavisor or any of the containers can consume it.
The list of user metadata is persistent on disk, can be managed locally by mgmt containers. Their values will be deleted or overwritten by the list stored in the cloud in remote mode.
Check out the list of default user metadata consumed by Pantavisor.
Config¶
Stores the PH_
prefixed keys of the configuration. This includes the Pantacor Hub client credentials.
These configuration keys are persistent. By default, they are saved in plain text, but this can be changed if set to a volume assigned to a crypt disk.
Disks¶
Pantavisor offers a way to define the physical storage medium by setting up disks and disk v2.
There are currently 4 disks types supported for disk:
- Non-encrypted directory
- Device Mapper crypt using without hardware acceleration (versatile)
- Device Mapper crypt using i.Mx CAAM
- Device Mapper crypt using i.Mx DCP
The new disk v2 supports all the previous ones and adds 2 new types:
- Swap disk based on zram
- Volume disk based on zram
Each disk, with the exception of swap disk, that is defined in the state JSON can then be privately used by the containers. They can also be internally used by Pantavisor volumes, as in the case of metadata.
Volumes¶
Additional volumes can be specified to be used internally by Pantavisor. These are some special cases that can be configured as volumes:
- pv--devmeta: if defined, Pantavisor will use this volume to save and load device metadata.
- pv--usrmeta: if defined, Pantavisor will use this volume to save and load user metadata.
- pv--phconfig: when defined, Pantavisor will reserve this volume to save and load Pantacor Hub credentials. To make Pantavisor use it, it has to be forced from configuration with the key
PV_STORAGE_PHCONFIG_VOL
.
Logs¶
Pantavisor can centralize all of your container logs, separated by revision, in one place on-disk. It does so by running a small server (Log Server) which offers a couple of sockets to the containers:
Output types¶
- pv-ctrl-log: to send log traces.
- pv-fd-log: to suscribe file descriptors.
There are a number of parameters that can be tweaked from the configuration that will affect Log Server, such as log.capture
, log.maxsize
, log.level
, etc. Between those, there is an important parameter called log.server.outputs
that allows to change how the logs will be stored:
These options are not mutually exclusive and can be combined in any fashion.
File tree¶
This is the default option. In this case the log output will be delivered in different files by containers. Pantavisor logs will also be stored in its own directory.
Single file¶
With this option, all logs will be centralized in one file. These logs are delivered in a JSON following this structure:
{
"tsec":9,
"tnano":0,
"plat":"pantavisor",
"lvl":"DEBUG",
"src":"logserver",
"msg": "starting logserver loop"
}
Standard Output¶
Outputs logs into stdout, which can be useful for debugging. Unlike stdout_direct, this sink is processed by the log server. This will provide more control on the output than its direct counterpart and a way of working that is more similar to the other sinks.
Note
In embedded mode, this is the same as sending the logs to dmesg. You can set ignore_loglevel and printk.devkmsg=on in cmdline to get all messages on console.
Standard Output Pantavisor¶
Same as Standard Ouput but filtering all messages except Pantavisor ones.
Standard Output Containers¶
Same as Standard Ouput but filtering all messages except container ones.
Standard Output Direct¶
Outputs logs into stdout. Unlinke stdout, this sink is not processed by the log server. This means the logs will be directly printed without going through the server. It will have a faster response and will provide logs during device initialization and teardown, when plain stdout would not be available.
This sink will be automatically enabled when stdout is enabled and log server is not available. That is, before its initialization and after its shutdown. Furthermore, it will always let pass all FATAL level log messages through.
NULL Sink¶
Send logs to /dev/null.
Timestamp format¶
All the log outputs, except for the NULL Sink, will print the timestamp along the log lines. It is possible to configure the format of the timestamp with the configuration key log.<output>.timestamp.format
, which can be set with the following values:
value | example |
---|---|
golang:Layout | "01/02 03:04:05PM '06 -0700" |
golang:RubyDate | "Mon Jan 02 15:04:05 -0700 2006" |
golang:ANSIC | "Mon Jan _2 15:04:05 2006" |
golang:RFC822Z | "02 Jan 06 15:04 -0700" |
golang:RFC1123Z | "Mon, 02 Jan 2006 15:04:05 -0700" |
Those formats are based in the Golang time constants. For other formats, the strftime formatters can be used setting the strftime:
prefix. For example: strftime:%d, %T %Y
. Please check the strftime manual for more information about formatters.
Trails and objects¶
Pantavisor trails, with their state JSONs are stored for each installed revision. The rest of artifacts that can be shared between revisions, are called objects and are stored separately.
Integrity¶
Pantavisor offers secureboot, a security mechanism to ensure integrity of the artifacts that are part of a revision. As revisions are composed of a state JSON and a series of artifacts pointed by that JSON, Pantavisor can perform a double level validation: artifact checksum and state signature.
Artifact Checksum¶
Objects and JSONs belonging to the revision that is about to be run after booting up are checked against the checksums stored in their state JSON by default.
As running a full SHA hashsum of all artifacts can be time consuming for low spec targets and thus increas boot up time, a lazy integrity validation on container storage can be performed using handlers, which are specified at the state JSON. Handlers are scripts to mount/unmount volumes. The only supported handler so far is dm-verity.
Checksum can also be totally disabled at the configuration level if boot up speed needs to be boosted further.
State Signature¶
Additionally to artifact checksum, the state JSON can be signed. This can be done from your host computer or from an automated CI, and it is then validated from Pantavisor using a cryptographic hash algorithm. It can be tuned according to one of these three levels of severity:
disabled
: no signature validation will be performed.audit
: all validations from lenient and strict will be performed, errors will be reflected in log but nothing will fail.lenient
: only the signed artifacts in the revision will be validated. Enabled by default.strict
: all artifacts in the revision must be signed and will be validated.
Validation is performed in Pantavisor at two points:
- when booting up, right after Pantavisor is initialized and before the current revision is loaded. In case of failure, the update will be reported as ERROR if possible and the device will either rollback or just reset.
- when a new remote update is received or a new local update is run. In case of failure, the update will be reported as WONTGO.
In either case, Pantavisor will parse all signatures, parse the protected JSON, calculate the hash depending on the elements included in the JSON and compare them with the signatures.
Right now, RS256, ES256, ES384 and ES512 algorithms are supported and Pantavisor must know the public key either by directly storing it on disk or by using a certificate chain.
On-disk public key¶
Note
We recommend to use the certificate chain of trust for your field devices.
In this option, Pantavisor will need the RSA public key to be stored on disk. You can store it during build time from the ca/ directory.
It will need the signature and information about how to generate it in the revision checkout. We offer pvr sig for convenience.
Certificate chain of trust¶
In this case, the public key will travel in the revision checkout in a x509 certificate. The certificate will be validated by Pantavisor against a root certificate located on disk .
The root certificate is placed during Pantavisor build. The build system will extract the files from the ca/ directory.
Depending on the subject CN of the signature certificate, we validate it with the following criteria:
- If the subject CN is equal to the value set in
PV_OEM_NAME
from your configuration and the signature does not includes any artifact in the BSP or OEM configuration directories, as those are can only be secured by the default root certificate, we validate it against the OEM root certificate. - For every other case, the signature is validated against the default root certificate.
It will also need the signature and information about how to generate it in the revision checkout. For this, you can use pvr sig command.
Garbage Collector¶
The Pantavisor garbage collector is in charge of cleaning up the /storage by automatically removing unused Pantavisor artifacts.
It works by removing logs, trails and stored disks that belong to old revisions. After all this, all orphan objects that were linked to removed revisions are deleted too.
The garbage collector will not affect any of the files related to the running revision, the revision that is being updated or the latest DONE revision. These revisions that are not affected by the garbage collector can be expanded to the factory revision using the PV_STORAGE_GC_KEEP_FACTORY configuration parameter.
Currently, it can be triggered by three different events:
- by a remote update that requires more disk space than available
- if a threshold of disk usage is surpassed
- with a command
Remote update¶
If a remote update requires more disk space than available, the garbage collector will be activated. This can be adjusted with the configuration PV_STORAGE_GC_RESERVED parameter.
Note
This type of trigger will not work with local updates. For those, you will need to use one of the following two options.
Threshold¶
Disk usage will be checked periodically and garbage collector will be activated if that threshold is reached. By default it is disabled. This can be changed with the PV_STORAGE_GC_THRESHOLD parameter.
If an object is put using the objects endpoint, the threshold will be disabled temporarily, which is done to avoid removing objects that could be linked to the upcoming new revision. This parameter can be changed from the configuration.
On Demand¶
Finally, our garbage collector can be triggered on demand issuing a command through Pantavisor control socket.