Skip to content

SCRAPING YAHOO FINANCE FOR ALL STOCK DATA

I found a great little snippet of code over on Kaggle,


and needed to build a system to run it from
https://tacticalware.com/jupyter-installation-on-a-headless-raspberry-pi-4-running-ubuntu-20-10/

Once you build the system as I did in my guide above you will need to do one other thing to get it to run
Open Putty
SSH to your Jupyter Pi
Login
Type:
sudo bash
pip instal yfinance

Now on another computer you can open your Jupyter notebook at
http://Jupyter:8888

Then go back to


Click File
Click Download
It will give you a file with the extension of ipynb

You can then go back to your Jupyter notebook and upload it

Once you upload it, run the notebook and it should scrape about 2.5GB of data, direct for your viewing pleasure

Hardware that I used:
Raspberry Pi 4 (4gb)
https://amzn.to/3q551IO

Plugable USB C to M.2 NVMe Tool-free Enclosure
https://amzn.to/3lflV3L

CanaKit 3.5A Raspberry Pi 4 Power Supply (USB-C)
https://amzn.to/3fNTYPu

CanaKit Raspberry Pi 4 Micro HDMI Cable – 6 Feet
https://amzn.to/33u5hr9

Western Digital 500GB WD_Black SN750 NVMe
https://amzn.to/3nZ5pH4


JUPYTER INSTALLATION ON A HEADLESS RASPBERRY PI 4 RUNNING UBUNTU 20.10

More and More, I am finding my way to Jupyter Notebooks. And guides online are scarce to come by, especially when you are running it on a Raspberry Pi 4. Below is a mixture of a few guides that I had found, and this is what worked for me.

For my setup I am running a Raspberry Pi 4 from a NVME M.2 Drive connected via USB 3.0. I am NOT using the slow micro SD card. To have the same setup as me, follow this guide:
https://tacticalware.com/boot-raspberry-pi-4-from-m-2-usb-drive/

After your Pi recognizes the M.2 NVME Drive, I then use Raspberry Pi Imager to write Ubuntu Server 20.10 to the USB Drive. Then I plug it into the Pi, to get ready for the steps below

Now boot your Raspberry Pi 4 and Login
Then
sudo bash
apt-get update
apt-get dist-upgrade
apt-get install net-tools
sudo hostnamectl set-hostname Jupyter
reboot
sudo bash
sudo rm /usr/bin/python
sudo ln -s /usr/bin/python3 /usr/bin/python
sudo apt-get install python3-pip
sudo pip3 install –upgrade pip
sudo apt-get install npm
sudo npm install -g configurable-http-proxy
sudo -H pip3 install notebook jupyterhub
jupyterhub –generate-config
sudo mv jupyterhub_config.py /root
nano /root/jupyterhub_config.py
ctrl +w
http://:8000
change to http://:8888
uncomment and delete space on this line c.JupyterHub.bind_url = ‘http://:8888’
ctrl +x
y
jupyterhub -f /root/jupyterhub_config.py
nano /lib/systemd/system/jupyterhub.service
[Unit]
Description=JupyterHub Service
After=multi-user.target

[Service]
User=root
ExecStart=/usr/local/bin/jupyterhub –config=/root/jupyterhub_config.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

ctrl + x to exit
y to save

sudo systemctl daemon-reload
sudo systemctl start jupyterhub
sudo systemctl enable jupyterhub
sudo systemctl status jupyterhub.service

on a different computer
open an internet browser
http://Jupyter:8888
login as any user

Back on the Raspberry Pi
sudo -H pip3 install testresources
sudo -H pip3 install jupyterlab
jupyter serverextension enable –py jupyterlab –system
nano /root/jupyterhub_config.py
uncomment and modicy the c.Spawner.default line to reflect the following:
c.Spawner.default_url = ‘/lab’
ctrl + x to exit
y to save
sudo apt-get install libatlas-base-dev
y
sudo -H pip3 install numpy
pip3 install seaborn
pip install Cython numpy
pip install cmake

sudo addgroup jupyter_admin
sudo adduser tacticalware
sudo usermod -aG jupyter_admin tacticalware
nano /root/jupyterhub_config.py
ctrl +x to exit
y to save
Add this to the end of the file
c.PAMAuthenticator.admin_groups = {‘jupyter_admin’}
ctrl + x to exit

You are now finished. On the other computer
Open an internet browser
http://Jupyter:8888
And play!

Hardware that I used:
Raspberry Pi 4 (4gb)
https://amzn.to/3q551IO

Plugable USB C to M.2 NVMe Tool-free Enclosure
https://amzn.to/3lflV3L

CanaKit 3.5A Raspberry Pi 4 Power Supply (USB-C)
https://amzn.to/3fNTYPu

CanaKit Raspberry Pi 4 Micro HDMI Cable – 6 Feet
https://amzn.to/33u5hr9

Western Digital 500GB WD_Black SN750 NVMe
https://amzn.to/3nZ5pH4

INSTALL MONGODB ON RASPBERRY PI 4

By far the best guide I have found, in starting this journey, was written by Mark Smith on https://developer.mongodb.com/how-to/mongodb-on-raspberry-pi/

If you want to see why things work the way they do, please visit his guide. My guide below is just the commands, and how I am connecting into the database once it is setup.

To begin, first you will want to install Ubuntu Server 20.10 as described here:
https://tacticalware.com/install-ubuntu-server-20-10-64-bit-on-raspberry-pi-4-using-a-m-2-drive/

Once you are up and running, ssh into the Raspberry Pi
Login

From the terminal run the following:
sudo bash

wget -qO – https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add –

echo “deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse” | tee /etc/apt/sources.list.d/mongodb-org-4.4.list

apt-get update

apt-get install -y mongodb-org

systemctl daemon-reload

systemctl enable mongod

systemctl start mongod

systemctl status mongod

CTRL button +c to exit

mongo
use admin

db.createUser( { user: "admin",
 
        pwd: "SuperSecretPassword",

        roles: [ "userAdminAnyDatabase",

                 "dbAdminAnyDatabase",

                 "readWriteAnyDatabase"] } )

exit

nano /etc/mongod.conf

Add these lines to the end of the file

security:

   authorization: enabled

CTRL +x to Exit
Y to Save

systemctl restart mongod

mongo

db.adminCommand({listDatabases: 1})

Make sure you had no errors

exit

mongo -u “admin” -p

Enter your admin password for mongo, aks your SuperSecretPassword

Type
db.adminCommand({listDatabases: 1})
This will list out your databases and ensure you created it properly

exit

nano /etc/mongod.conf
and Change the bind ip from 127.0.0.1 to 0.0.0.0

net:
   port: 27017
   bindIp: 0.0.0.0

CTRL +x to Exit
Y to Save

systemctl restart mongod

ufw allow 27017/tcp

MongoDB is now successfully setup and running. On to configuring the Windows 10 system

To connect to the MongoDB from a Windows 10 computer

Open an Internet Browser on your Windows 10 computer
Navigate to https://robomongo.org/
Download the Robomongo Robo 3T Client
Once you install it, add these settings to connect to your system
Click Create
Connection Tab
Type – Direct Connection
Address – The IP of your Raspberry Pi
Port 27017
Authentication Tab
Check mark Perform Authentication
Database is admin
User Name is admin
Password is your SuperSecretPassword
Auth is SCRAM
Click Save
Click Connect

You are now connected and ready to run with your MongoDB environment

Hardware that I used:
Raspberry Pi 4 (4gb)
https://amzn.to/3q551IO

CanaKit 3.5A Raspberry Pi 4 Power Supply (USB-C)
https://amzn.to/3fNTYPu

CanaKit Raspberry Pi 4 Micro HDMI Cable – 6 Feet
https://amzn.to/33u5hr9

Western Digital 500GB WD_Black SN750 NVMe
https://amzn.to/3nZ5pH4

Plugable USB C to M.2 NVMe Tool-free Enclosure
https://amzn.to/3lflV3L

INSTALL UBUNTU SERVER 20.10 64-Bit ON RASPBERRY PI 4 USING A M.2 Drive

This guide shows you how to install an Operating System on a Raspberry Pi 4. The one I chose for this guide is Ubuntu Server 20.10 64 bit, so if you are installing that, you can follow along easily. If you are wondering which hardware I used, the exact items are linked at the bottom

On a Windows computer
Insert your USB M.2 Drive into the USB 3.0 Port on your Windows computer
Open a web browser
Navigate to
Download the Raspberry Pi Imager

Once it is downloaded, double click the imager.exe file
Select Yes
Click Install
Checkmark Run Raspberry Pi Imager
Click Finish

Now that the Raspberry Pi Imager is running
Click Choose OS
Scroll down and select Ubuntu
Then select Ubuntu Server 20.10 (RPI 3/4/400) 64-BIT
Select your SD Card (500GB WD M.2 Drive)
Click Write
All existing data …. will be erased. Are you sure you want to continue?
Click Yes
Data is written to the SD Card
The data is automatically verified after it is written
Once it finishes you will see the message that “you can now remove the SD card from the reader”
Remove the card
Click Continue

Now back on the Raspberry Pi
Plug in network cable
Connect the keyboard
Plug in a mouse
Connect the HDMI cable
Install the M.2 USB Drive you just made (up above)
Plug in the power (Last step)

Boot up your Raspberry Pi
The first time you login the username is ubuntu
The password is ubuntu
You will be forced to change the password immediately
Now you will want to run the following commands
sudo bash
apt-get update
apt-get dist-upgrade
y
apt install net tools
ifconfig
Note your ip address

You can now connect remotely and finish setting up the system

You are done. Your Raspberry Pi 4 is setup with the Ubuntu Server 20.10

Hardware that I used:
Raspberry Pi 4 (4gb)
https://amzn.to/3q551IO

SanDisk 32GB Ultra microSDHC UHS-I Memory Card with Adapter
https://amzn.to/2Vfvo0y

CanaKit 3.5A Raspberry Pi 4 Power Supply (USB-C)
https://amzn.to/3fNTYPu

CanaKit Raspberry Pi 4 Micro HDMI Cable – 6 Feet
https://amzn.to/33u5hr9

Western Digital 500GB WD_Black SN750 NVMe
https://amzn.to/3nZ5pH4

Plugable USB C to M.2 NVMe Tool-free Enclosure
https://amzn.to/3lflV3L

Thanks for reading…….

COMBINE MULTIPLE CSV FILES FROM THE COMMAND LINE AND RETAIN FILENAMES

For this guide we will take thousands of csv files and combine them into One single file, which will be uploaded to a database. This file will have a new column inserted, where the originating filename will be added next to the row. In scraping sites, often my output files are separated and after I clean them up I need to combine them. This is how I do it.

On my Windows 10 computer
I will copy my CSV files over to a secondary drive, leaving the originals intact.
Then I will open a command prompt as an administrator and navigate to that drive. If the files are on my H: drive will navigate by typing
H:
Then press Enter

Now I will navigate to the “dataset” folder
cd dataset

I will then list the directory contents to ensure this holds my csv files
dir

Once I see all of my csv files I will enter the following:

for /f %a in ('dir /b *.csv') do for /f "tokens=*" %b in (%a) do echo %b,%a >> combined-date.csv

Then press Enter

This will create a file called combined-04132021.csv (which is today’s date)

And I can then upload that file to my database easily

COMBINE MULTIPLE CSV FILES FROM THE COMMAND LINE

For this guide we will take thousands of csv files and combine them into One single file, which will be uploaded to a database. In scraping sites, often my output files are separated and after i clean them up I need to combine them. This is how I do it.

On my Windows 10 computer
I will copy my CSV files over to a secondary drive, leaving the originals intact.
Then I will open a command prompt as an administrator and navigate to that drive. If the files are on my H: drive will navigate by typing
H:
Then press Enter

Now I will navigate to the “dataset” folder
cd dataset

I will then list the directory contents to ensure this holds my csv files
dir

Once I see all of my csv files I will enter the following
copy *.csv combined-date.csv
Then press Enter

This will create a file called combined-04132021.csv (which is today’s date)

And I can then upload that file to my database easily

THE VESPER

  • General

“Three measures of Gordon’s, one of vodka, half a measure of Kina Lillet. Shake it very well until it’s ice-cold, then add a large thin slice of lemon peel. Got it?”

-James Bond

US Stock Market Terminology

Quick Reference for Stock Market Terms. When you hear the major terms including equities, derivatives, fixed income, and cash, the information below are what constitutes each market

Equities
Stock
ETF
Mutual Funds

Derivatives
Options
Commodities
Forex (currencies)

Fixed Income
Bonds
Treasuries
Certificate of Deposit

Cash
Savings
Money Market

PAYPAL CRYPTO – CONVERTING CASH TO BITCOIN

  • Crypto

Recently Paypal has launched their crypto system, allowing you to purchase and sell crypto. Here is the rundown on how to do it, and what it does not do when converting cash to bitcoin, ethereum, litecoin and others

To convert cash to Bitcoin
Open a browser and navigate to https://www.paypal.com
Login to your Paypal account

Side Note:
Make sure your Paypal account is connected to your checking account already.. If it is not, you will have to do this

Once logged into Paypal, click on the Crypto button at the top right
Click BUY next to Bitcoin
Select the Big Blue Buy button at the center again

Side Note:
If this is your first time buying crypto with Paypal you will need to Confirm some extra info before you buy

Next you will put your amount listed in USD, into the Paypal window
Paypal will also add their Fees on top of the amount you specified
Click Confirm, and your crypto will be purchased with Paypal.

You now own crypto

Now for my issues with it
1. Paypal does not let you transfer your crypto out. They hold your crypto, and you hostage to their HUGE fees.

2. You do not have control over your crypto. They get hacked, you lose your crypto. You can do nothing about it

3.They show a chart that says Bitcoin is $59,133, however when you go to sell it, they immediately take roughly half a percent off. So in trying to sell it you see Bitcoin is $58,823. Cancel out of that screen, and Bitcoin is magically back at $59,151. Go back into the sell screen and your half a percent or so is gone again. Over and over I did this, and over and over the spread didn’t make sense.

4. When selling you also have to pay a large paypal fee

5. Then to transfer your cash back into your account, you pay another 1%

All in all, on a $10,200 transaction, I paid $410.33 in fees immediately and will be hit by the other 1% when it goes to my bank. Bitcoin did not decrease in price either during my 10 minute test. It actually increased in value around $80.