Paul's Mutterings and Meanderings

Saturday, 1 January 2022

Your Own Cloud Server

So many different cloud storage solutions out there such as Dropbox, Google Drive, One Drive and so on. But what if you want to manage all your data yourself, in the comfort of your own home, NextCloud could be just what you're looking for.

In this article, I'm going to go through how to setup your own cloud server at home, you need an install of Linux, I've used Ubuntu Mini for this, you need to setup a static IP for the computer that is going to be the cloud server and you also need to forward ports 80 and 443 to that IP address. If you can't install Linux and don't know how to forward the ports then this isn't for you.

The only thing you really need to do during the installation is enable the ssh server so you can do the rest remotely.

The computer I'm using is a small form factor machine with a 250GB SSD and an external RAID enclosure with 2 x 4TB drives connected by USB

Plug the RAID device and the run dmesg shows the RAID device on my machine at /dev/sda

As it's a big drive we need to partition using

sudo parted /dev/sda

mklabel gpt

mkpart ext4 0% 100%

quit

Now time to format it with

sudo mkfs.ext4 /dev/sda

This will give you an output similar to this:

Found a gpt partition table in /dev/sda

Proceed anyway? (y/N) y

Creating filesystem with 976740352 4k blocks and 244187136 inodes

Filesystem UUID: 8da4f15d-2def-4cb9-ab7f-c972479529bd

Superblock backups stored on blocks:

32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,

102400000, 214990848, 512000000, 550731776, 644972544

Allocating group tables: done

Writing inode tables: done

Creating journal (262144 blocks): done

Writing superblocks and filesystem accounting information: done

Copy that UUID number, it will be different for you, we will now be modifying the fstab file with

sudo nano /etc/fstab

Add a line like this to the end, using your UUID.

UUID=8da4f15d-2def-4cb9-ab7f-c972479529bd /mnt/Data ext4 defaults,noatime 0 1

Then CTRL X then Y and enter to save.

Make the mount point with;

sudo mkdir /mnt/Data

Then run sudo mount -a to mount the drive.

If you run df -h you should see something like this which shows the RAID drive up and running.

Filesystem Size Used Avail Use% Mounted on

udev 3.9G 0 3.9G 0% /dev

tmpfs 786M 820K 785M 1% /run

/dev/sdb5 228G 3.9G 213G 2% /

tmpfs 3.9G 0 3.9G 0% /dev/shm

tmpfs 5.0M 0 5.0M 0% /run/lock

tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup

/dev/sdb1 511M 4.0K 511M 1% /boot/efi

tmpfs 786M 0 786M 0% /run/user/1000

/dev/sda 3.6T 89M 3.4T 1% /mnt/Data

We're now going to create two directories, one for the Apache web server and one for the NextCloud data.

sudo mkdir /mnt/Data/www

sudo mkdir /mnt/Data/NextCloud

Now we create a symbolic link in the /var directory which points to the www directory we just created.

ln -s /mnt/Data/www/ /var/www

For now we are just going to set the permissions to 0777 with:

chmod 0750 -R /mnt/Data

Now it's time to start installing the required software to get our cloud server running.

sudo -s

apt update && apt upgrade

apt install ssh screen apache2 php mariadb-server php-fpm php-pear php-gd php-mysql php-redis php-curl php-json php-mbstring unrar lame mediainfo subversion ffmpeg redis software-properties-common mc net-tools unzip ufw curl mc php-zip -y

Time to secure the database with:

mysql_secure_installation

You should then see the following

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB

SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current

password for the root user. If you've just installed MariaDB, and

you haven't set the root password yet, the password will be blank,

so you should just press enter here.

Enter current password for root (enter for none):

OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB

root user without the proper authorisation.

You already have a root password set, so you can safely answer 'n'.

Change the root password? [Y/n] n

... skipping.

By default, a MariaDB installation has an anonymous user, allowing anyone

to log into MariaDB without having to have a user account created for

them. This is intended only for testing, and to make the installation

go a bit smoother. You should remove them before moving into a

production environment.

Remove anonymous users? [Y/n] y

... Success!

Normally, root should only be allowed to connect from 'localhost'. This

ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] y

... Success!

By default, MariaDB comes with a database named 'test' that anyone can

access. This is also intended only for testing, and should be removed

before moving into a production environment.

Remove test database and access to it? [Y/n] y

- Dropping test database...

... Success!

- Removing privileges on test database...

... Success!

Reloading the privilege tables will ensure that all changes made so far

will take effect immediately.

Reload privilege tables now? [Y/n] y

... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MariaDB

installation should now be secure.

Thanks for using MariaDB!

Now we create the database for the cloud server with the following, replace password with a good strong password of your own.

mysql -u root -p

CREATE USER 'nextcloud'@'localhost' IDENTIFIED BY 'password';

GRANT ALL PRIVILEGES ON nextcloud.* TO 'nextcloud'@'localhost' WITH GRANT OPTION;

exit

Little bit of file editing now with the following :

nano /etc/mysql/conf.d/mysql.cnf

Add this to the bottom of the file and then save it:

[mysqld]

group_concat_max_len=8192

innodb_flush_log_at_trx_commit = 2

nano /etc/php/7.4/apache2/php.ini

Change the following lines to match shown below, the time zone one should be changed for your part of the world.

max_execution_time = 120

memory_limit = -1

date.timezone = Europe/London

nano /etc/php/7.4/cli/php.ini

Change the following lines to match shown below, the time zone one should be changed for your part of the world.

max_execution_time = 120

date.timezone = Europe/London

Time now to make the config file for the apache web server:

nano /etc/apache2/sites-available/nextcloud.conf

Add this in changing the ServerName and ServerAlias to your server name, you can use a dynamic host such as no-ip or duckdns to get an domain name, I use my NAS drive to provide one.

ServerName your_domain

ServerAlias www.your_domain

Options FollowSymLinks

AllowOverride All

Order allow,deny

allow from all

</Directory>

DocumentRoot /var/www/nextcloud

</VirtualHost>

Let's get the Nextcloud software installed:

cd /var/www

wget https://download.nextcloud.com/server/releases/nextcloud-23.0.0.zip

unzip nextcloud-23.0.0.zip

rm nextcloud-23.0.0.zip

Set up permissions

chmod 0750 -R /var/www/

chmod 0750 -R /mnt/Data/NextCloud/

chown -R www-data:www-data /mnt/Data/NextCloud/

chown -R www-data:www-data /var/www/

Setup the firewall rules now with:

ufw default deny incoming

ufw default allow outgoing

ufw allow ssh

ufw allow 80

ufw allow 443

ufw enable

Time now to activate the server config

a2dissite 000-default.conf

a2ensite nextcloud.conf

a2enmod proxy_fcgi setenvif

a2enconf php7.4-fpm

a2enmod rewrite

systemctl restart php7.4-fpm

systemctl restart apache2

systemctl restart mysql

Now to setup some ssl security

apt install certbot python3-certbot-apache

Then just type certbot followed by enter.

And answer the questions.

On the screen that says "Which names would you like to activate HTTPS for" you should see the names of the ones you entered in the Apache config file earlier. Just press enter here.

And then select 2 to create a redirect.

That so far is all the installation done, now it's time to open up a web browser and point it to the domain you created for the cloud server, this is the dynamic dns one.

http://servername:

Fill in all the details requested and make sure you change the data path to

/mnt/Data/NextCloud or whatever you have set yours to.

Once that's all done you can install the desktop client and the mobile client and have a play around. That's all your data under your control.

Saturday, 17 April 2021

PiCorder (The Next Generation)

The original PiCorder worked, took ages to start recording and I couldn't hear what I was recording as it was being recorded, it was time for a rethink.

I decided to throw away the original idea and add a small USB sound card, this is what I got in all for it, apart from the Raspberry Pi Zero W of course.

These are fantastic, reversible USB micro leads, not USB C, USB micro, they can go in either way around.

I got them from Amazon, £7.99 for 2, there are probably other places so please shop around/

This is the mic I got, again from Amazon, £7.59, you can probably use any mic with a 3.5mm jack, I didn't have one so this did the job.

Nice little case for it all to go in, as it's transparent you can use the onboard LED as a sort of status thingy, Amazon again and £8.99

Then there's the sound card, there's quite a few of these about, I got two and they seem to work quite well to be fair. Again from Amazon at a cost of £7.59.

You don't have to get one of these but in my mind it makes the sound card fit better with the case. Cost for the two is £5.10 from Amazon.

Power was the next thing on my shopping list, I got one of these, mainly because I liked the look of it and if you press the test button on it then it turns the power back on, more on that later. I managed to get one from Morrisons at £9.99.

And last but not least apart from the Raspberry Pi itself was something to give me a bit of directionality without being too big and intrusive and was light. I looked at bullet mics and parabolic dishes and although the dish itself probably gives the best results I fell back on the idea of the really old hearing aids, basically a funnel. I found these in Poundland and managed to get some grey ones.

You need a micro SD card too of course, I got a 32GB one, I would suggest getting a decent make, I got some cheap ones and they gave up after about a week.

Put together will all the magic stuff from a glue gun and the old favourite Blu Tack and it looks like this, the special effect guys from Blakes 7 couldn't have done better. You have a headphone socket so that you can hear what you're recording and you can collapse the funnel back to make it less obtrusive. It's not brilliant but it works, the funnel does get rid of some outside noise and I'll include some sound samples later on.

I also added some small push buttons, top one is connected to GPIO 3 and ground, the bottom to GPIO 13 and ground. I soldered these directly to the board but you can break it and the warranty is void of course.

Time to put it all together now, I'm not going through how to install the OS onto the SD card, I used Raspbian Lite for it, don't really need a desktop environment and it's all set up by SSH so make sure that's enabled.

We're going to have to get the sound card working first and change the playback and sound levels, run the following:

sudo nano /usr/share/alsa/alsa.conf

defaults.ctl.card 0

defaults.pcm.card 0

Change these to a 1 as shown below.

defaults.ctl.card 1

defaults.pcm.card 1

Run sudo alsamixer and you should see this, select the Speaker and Mic and up the levels and we are now ready to get on with the rest.

We're going to turn off bluetooth as this seemed to be the cause of some electrical noise.

Open up the /boot/config.txt file with:

sudo nano /boot/config.txt

Add the following to the bottom of the file

# Disable bluetooth

dtoverlay=disable-bt

# Allows power off and on

dtoverlay=gpio-shutdown

Now we remove the bluetooth software

sudo systemctl disable hciuart.service

sudo systemctl disable bluealsa.service

sudo systemctl disable bluetooth.service

sudo apt-get purge bluez -y

sudo apt-get autoremove -y

Now reboot to continue, once it's back up you should find a press of the top button will shut the Pi down and once it's closed down completely the status lights on the charger will go off. Press the test button on the charger and the Pi will come back on.

I've made a small script called mtbootme which is in the bin folder, makes it easier to add things when it boots up, just run sudo nano /bin/mtbootme and add the following lines

# Turns the light off

echo 0 >/sys/class/leds/led0/brightness

# Makes the light flash

echo heartbeat >/sys/class/leds/led0/trigger

Save it and make it executable with sudo chmod +x /bin/mtbootme

Now we need a way of recording the sound and also of being able to monitor it while it's recording. There's probably a better way of doing it but I came up with this, there's about a second delay.

The bit of shell script I use checks for a file number, if it's not there it starts at 1, if it is there it adds a 1 to it.

sudo nano /bin/mtcapture

Paste this lot in

#!/bin/bash

# shuts down the wifi

ifconfig wlan0 down

# Turns the LED on fully

echo 0 >/sys/class/leds/led0/brightness

echo 1 >/sys/class/leds/led0/brightness

# sorts the file number

FILE=/share/filenum

if [ -f "$FILE" ]; then

echo "$FILE exists."

else

echo "$FILE does not exist."

echo "0"> /share/filenum

read newfile < /share/filenum

echo $newfile

newfile=$((newfile+1))

echo $newfile > /share/filenum

# Records the noises

arecord -f cd - | tee /share/AudioTux1-$newfile.wav | aplay - &

echo $newfile

# updates the file number

echo $newfile > /share/filenum

Make it executable with sudo chmod +x /bin/mtcapture

That's how we start it recording, don't run this just yet as it will turn the wifi off and you'll lose your connection or rem out the wifi line while testing.

Now we do the stop recording file

sudo nano /bin/mtstop

#!/bin/bash

# Kill all recording services

pkill mtcapture

pkill arecord

pkill aplay

# Starts up the wifi

ifconfig wlan0 up

# Flashes the green light so we know recording has stopped

echo 0 >/sys/class/leds/led0/brightness

echo heartbeat >/sys/class/leds/led0/trigger

Save it and make it executable again with sudo chmod +x /bin/mtstop

A bit of python now to monitor the stop start button.

sudo nano record.py

This lot goes in here.

# !/bin/python

import RPi.GPIO as GPIO

import time

from subprocess import call

import os

GPIO.setmode(GPIO.BCM)

GPIO.setup(13,GPIO.IN,pull_up_down=GPIO.PUD_UP)

flag = False

while True:

input = GPIO.input(13)

if(input == False):

if(flag == False):

os.system("/bin/mtcapture")

flag = True

time.sleep(0.4)

else:

os.system("/bin/mtstop")

flag = False

We don't need to make this executable to run it.

Finally we edit the rc.local file to get things working on startup

sudo nano /etc/rc.local

Add these two lines just before the exit 0 at the bottom

python2 /root/record.py &

/bin/mtbootme &

Save it and reboot and you're ready to go, one last thing I've done is to install samba so I can pull the files off it. All the recordings are saved in the directory /share.

Install Samba and MC, it's one of the first things I install as it makes navigating the structure and editing really easy.

sudo apt install samba mc

Once done edit the samba config file

sudo nano /etc/samba/smb.conf

Add this to the bottom[Shares]

path = /share

public = yes

browseable = yes

writeable = yes

delete readonly = yes

force create mode = 777

force directory mode = 777

writeable = yes

directory mode = 777

create mode = 777

Save it and restart samba with sudo service smbd restart

And you are now ready to go, once you've powered it up the green status LED should be flashing, plus some headphones into the socket and press the lower button and it should start recording, you'll hear a delayed sound in the headphones, press it again and it will stop.

It's not supposed to be a professional bit of equipment, it's small, it's light and it sort of works and more than anything it's a bit of fun and it will get you more involved with nature.

Advantages of the funnel at the end are that it's completely collapsible, you can put it in a pocket and fold it over as shown below and it still works. Disadvantages are that you look like a mad professor when you're out and about with it.

And finally some sample sounds, there's a bit of noise in the background which is from a nearby industrial unit,

Sunday, 28 March 2021

PiCorder

I enjoy getting out and about when I can, nature reserves are a great place to go and I've thought a while about recording birdsong and other noises such as rivers and so on. I could use a phone but where's the fun in that and then once you get the phone out there's the distractions of social media. It's nice to leave those things behind.

I decided to knock up a sound recorder using a Raspberry Pi Zero W, it's never going to be brilliant, there's no display or controls on it so the sound is a fixed level but it had to be tried.

I got hold of one of these from Amazon, no drivers needed, just plug it in and it will work is what was said everywhere. Not quite really but in the end it wasn't too hard to get sorted.

I followed all the initial instructions and nothing happened, just an error so more searching on the Google thing found me this.

sudo nano /usr/share/alsa/alsa.conf

defaults.ctl.card 0

defaults.pcm.card 0

Change these to a 1 as shown below.

defaults.ctl.card 1

defaults.pcm.card 1

Save the file and it, sort of. Loads of noise on the recordings and suggestions of adding capacitors across the USB ports to dampen it down seemed drastic. Not a lot seemed to work, looked like it was going to be a complete failure.

So I had a poo and it came to me, first thing was get the recording level
up, I'm running everything here as root, there's nothing on here that is precious.

Run alsamixer and you should see this:

Press F4 to select the capture device and using the arrow keys increase the gain to the top as shown below.

Press escape to leave it.

As mentioned before, I was getting loads of noise when I tried this earlier, I decided to hard wire power and a record button onto the Pi, this meant soldering the power wires to the board on pins 2 and 6, you can see the complete list of pins here.

And the stop start button soldered to pins 36 and 34 (GPIO 16 and Ground) Don't blame me if you break your Pi while doing this.

I got hold of one of these so I could connect the mic into the Pi, no cables needed, again from them Amazon people.

Then I decided that as I wanted to keep power usage down I would disable the bluetooth connection and remove the bluetooth software.

Open the config.txt file with

sudo nano /boot/config.txt and then add this to the bottom of the file:

# Disable Bluetooth

dtoverlay=disable-bt

Save and close it then run the following to remove all the bluetooth stuff.

sudo systemctl disable hciuart.service

sudo systemctl disable bluealsa.service

sudo systemctl disable bluetooth.service

sudo apt-get purge bluez -y

sudo apt-get autoremove -y

I rebooted and carried out a test recording and there was no electrical noise apart from the odd regular buzz, I disabled the wifi and retested and this time no noise at all.

It was now time to stick it in a box and make it work.

I needed a way of getting the files off afterwards, easiest option was install samba with

apt install samba

Create a directory with

mkdir /share

Open the samba config with

nano /etc/samba/smb.conf

Add the following at the bottom

[Sounds]

path = /share

public = yes

writeable = yes

browseable = yes

Restart Samba with service smbd restart

I decided to create a sound recording service with:

nano /etc/systemd/system/soundrec.service

[Unit]

Description=Sound Recording

[Service]

ExecStart=/bin/mtcapture

WorkingDirectory=/share

StandardOutput=inherit

StandardError=inherit

Restart=always

User=root

[Install]

WantedBy=multi-user.target

I'm not enabling the service to auto start

Create some files in the bin folder:

nano /bin/mtstart

#!/bin/bash

# shuts down the wifi

ifconfig wlan0 down

service soundrec start

# sleep 1

echo 0 >/sys/class/leds/led0/brightness

# Makes the green LED flash so we know it's recording

echo heartbeat >/sys/class/leds/led0/trigger

nano /bin/mtstop

#!/bin/bash

service soundrec stop

# Starts up the wifi

ifconfig wlan0 up

# sleep 1

echo 0 >/sys/class/leds/led0/brightness

sleep 3

# Leaves the green LED on so we know it's finished

echo 1 >/sys/class/leds/led0/brightness

nano /bin/mtcapture

#!/bin/bash

for (( ; ; ))

FILE=/share/filenum

if [ -f "$FILE" ]; then

echo "$FILE exists."

else

echo "$FILE does not exist."

echo "0"> /share/filenum

read newfile < /share/filenum

echo $newfile

newfile=$((newfile+1))

echo $newfile > /share/filenum

arecord -f cd /share/AudioTux1-$newfile.wav

done

This increments the file number each time it's run so we can have as many files as we want until we run out of space.

You need to make all these executable with chmod +x filename

Then a little bit of python to get it working

nano record.py

# !/bin/python

import RPi.GPIO as GPIO

import time

from subprocess import call

import os

GPIO.setmode(GPIO.BCM)

GPIO.setup(16,GPIO.IN,pull_up_down=GPIO.PUD_UP)

flag = False

while True:

input = GPIO.input(16)

if(input == False):

if(flag == False):

print "Start"

os.system("/bin/mtstart")

flag = True

time.sleep(0.4)

else:

print "Stop"

os.system("/bin/mtstop")

flag = False

time.sleep(0.4)

Open up the rc.local file with

nano /etc/rc.local

And add the following just before the exit 0 at the bottom

python2 /root/record.py &

Then restart it, once it's up and running, pressing the button connected to the GPIO pins should start it recording. A second press will stop if, while it's recording the wifi will be disabled. If you want you can add a second button connected to pins 5 and 6 (GPIO 3 and Ground), add the following to the bottom of the /boot/config.txt file.

dtoverlay=gpio-shutdown

Reboot and on press on the button will shut the pi down, a second press will restart it.

Here's the finished item, it's not overly pretty as I got the wrong case but it works, the USB mic at the top can be unplugged and put on an extension cable and possibly fitted into a parabolic dish, but that's for another time.

It does work, seems to take ages to start recording as the service has to start and once started it then takes ages to stop the service.

It needed a rethink and so was born "PiCorder, the Next Generation"

Instructions are here.

Some more Pi

Been a while since I've done anything on here, let's face it 2020 was a washout and the first bit of 2021 has gone that way too. It does mean that things at home have come higher up the list of things to do and one of those things was to add a camera to a Raspberry Pi, point it at the sky and take pictures. The main idea was to get some shots of the stars moving but the clouds looked interesting too.

The first attempt was a quick lash up stuck in a plastic bag to give it some water proofing and put on the floor in the garden out of site of the street light at the back. This was a Raspberry Pi 2 and the version 1 camera which can go down to about a 6 second exposure, nothing lower than that so the stars didn't really stand out that well and there is a lot light pollution too.

Here's my first test, not great but it seems to show a bit of promise if we can get a clear night and even more so if we could try somewhere with a darker sky.

The other plan or project was a camera at the from of the house which apart from doing some timelapse would also take a picture every day at dawn, noon and sunset for a year from the 21st of March. For no other reason than it can be done. The noon bit was fairly straight forward, the sunrise and sunset were going to be harder. Until I came across something called "Heliocron". You can use this app to run a task at various times of the day and it calculates sunrise and sunset for you. Many thanks to the author for this.

Let's get started with my SkyCam as I call it, to run properly the Raspberry Pi needs a connection to a network so it can get the correct time. I have a plan B for a travel version but I'll come back to that.

I'm going to build this with a Raspberry Pi 3, a version 2 camera which gives 8Mp and shutter speeds down to 10 seconds. I need a waterproof case so I can stick it outside and a power and ethernet lead out of it. There will be some code which I've borrowed and modified from various searches, I can't remember where I got them but if it's yours then it's much appreciated.

This may not be the best way of doing it but it seems to work.

The finished article looks like this, I've managed to find an old Tripod bush so I can mount it outside and adjust the position.

Using the above mentioned Heliocron, it changes the exposure settings as it gets darker and lighter to keep some detail. I'm not clever enough to get it to take a measurement and do it automatically.

The script that will be taking the pictures is called:

mtshutter

There's going to be a couple of versions of it called mtday, mtdusk, mtearly and mtnight.

I stick the mt in front as it makes them easier to find.

I'm storing them in the /bin folder, I can't see any reason as to why not but if I'm wrong I'm sure somebody will tell me.

They all look fairly similar except for the picture taking settings, starting with mtshutter:

The pictures once taken are stored on a small file server mounted at /mnt/webcam in their own folder called skycam1, there may be another one in the future. You can of course change the location to wherever you want.

#!/bin/bash