backup sync

backup and syncing

It takes a lot of trials to understand the best setup to sync and backup our own work. Ideally we want to save everything and be able to find what we need in a second. I’m personally really concerned about environment and I decided not to use any cloud and minimize my data storage under 1TB. The idea is to remove the original files and move its content into these articles which are making sense of the work done. We protect some big files but we can’t share the information because we didn’t think how we could make the material shareable. Once you are able to answer few simple questions you can completely restructure your working flow in a much more efficient way.

decisions

There are different types of backups and it is important to find the most fitting for the work we do:

revision control

commit logic when we need to preserve the history of editing

databases

for the information we store in databases

cold storage

files we save in case something goes wrong

Once we have defined a backup strategy is then easier to start working with it by answering simple questions:

public/private

the way to handle private information is to remove those ones from the repository synced. There are enough way to use variables to avoid exposing sensitive information to the public.

constant edit

the files we need to keep updated

archive

what we didn’t use in a long time and we might not need for good (logs)

compression

can we compress the data (pictures)?

ideas vs data

can we separate the idea from the data which is usually sensitive?

recurrent tasks

Manage the proper storage is work. It requires a lot of organization and thinking but it brings as well different benefits. Some common tasks:

pictures

scroll pictures and delete similar ones

navigate

navigate folders and

automate labeling

LLMs can help a lot to organize and index the information we have.

current structure

My all records are structured in this way

pictures

personal pictures are in pers, the rest doesn’t contain any sensitive information

websites

the reduced images to publish

lav/

main folder with all the professional (and semi-professional) related projects

src/

source code synced with github

cios/

agricultural projects (GIS)

viudi/

music instrument making

kotoba/

lingustic

comp/

sensitive files connected with previous working experiences

music/

despite streaming some folk song I prefer to

VideoProd/

current video production, drafts are in in_prog/ and once the video is published it gets moved to pubblicati/ and the single clips get deleted forever

audio

collection of soundfonts and vst for daws. There are in a separated directory considering the big size of some soundfonts

zughi/

old roms (nes, snes)

du -sch ~/Pictures ~/lav/ ~/Music ~/VideoProd ~/Audio ~/Zughi

90G	/home/sabeiro/Pictures
21G	/home/sabeiro/lav/
33G	/home/sabeiro/Music
214G	/home/sabeiro/VideoProd
2.1G	/home/sabeiro/Audio
7.8G	/home/sabeiro/Zughi
365G	total

My all digital life is under 400GB and I can soon delete mode video clips, some old log-files and compress more images

sync

I manly use file:///home/sabeiro/syncUsb.sh and file:///home/sabeiro/syncFromUsb.sh to sync with external drives. Sync reads all the file diffs so it is important to zip folders we don’t touch often

DISKN=$1
if [ -z "$1" ]; then exit
fi
rsync -avz --delete ~/Pictures $DISKN
rsync -avz --delete ~/Music $DISKN
rsync -avz --delete ~/VideoProd $DISKN
rsync -avz --delete ~/Zughi $DISKN
rsync -avz --delete ~/lav $DISKN
rsync -avz --delete ~/Audio $DISKN
rsync -avz ~/videoOld $DISKN

revision control

When I started coding back in 2006 I was using subversion which I find slick. I find git less efficient but this is the standard. git conserves in the directory the revision database making packages huge. So I had to decide to organize my work in a more centralized way. The main move I made is to create a common documentation across all projects which is spiega/. This repo contains all the information across all projects and the conversion tools to publish it. The published pages converge in intertino.it which is synced with sys_sync.sh. All the images to publish

publication

Ideally most of the information can be used to publish articles and websites. I don’t expect any traffic but I still find essential to share any relevant experience.

du -sch ~/lav/siti/* | grep [M,G]

18M	/home/sabeiro/lav/siti/antani
51M	/home/sabeiro/lav/siti/anticolo
39M	/home/sabeiro/lav/siti/assets
16M	/home/sabeiro/lav/siti/blender_twin
15M	/home/sabeiro/lav/siti/cios
18M	/home/sabeiro/lav/siti/CV
142M	/home/sabeiro/lav/siti/dauvi
5.8M	/home/sabeiro/lav/siti/db
1.1G	/home/sabeiro/lav/siti/f
62M	/home/sabeiro/lav/siti/Fiziko
6.6M	/home/sabeiro/lav/siti/font
408M	/home/sabeiro/lav/siti/foto
9.8M	/home/sabeiro/lav/siti/geomadi
33M	/home/sabeiro/lav/siti/intertino
276M	/home/sabeiro/lav/siti/kotoba
34M	/home/sabeiro/lav/siti/perrizio.zip
63M	/home/sabeiro/lav/siti/phpsql
69M	/home/sabeiro/lav/siti/portfolio
17M	/home/sabeiro/lav/siti/roundcube
15M	/home/sabeiro/lav/siti/scritti
52M	/home/sabeiro/lav/siti/spiega
17M	/home/sabeiro/lav/siti/storia
24M	/home/sabeiro/lav/siti/storie
153M	/home/sabeiro/lav/siti/talk
18M	/home/sabeiro/lav/siti/tastal
30M	/home/sabeiro/lav/siti/tmp-www
95M	/home/sabeiro/lav/siti/v
1.7G	/home/sabeiro/lav/siti/viaggi
574M	/home/sabeiro/lav/siti/viudi
5.0G	total