Skip to content
Home » Scraping

Scraping

UPGRADE MAUTIC ON UBUNTU 20.04

Quick how-to on the way I upgraded mautic, since upgrading through the web breaks my installations every time.

Login into your Virtualmin

Backup all of your virtual servers

Download the backups to your pc

Open Putty

SSH to Server

sudo su –

cd to your mautic directory (/home/user/html)

php bin/console mautic:update:find

php bin/console mautic:update:apply

php bin/console mautic:update:apply –finish

Open your mautic installation, refresh the web page, check the bottom right, and you will see the latest version. In my case it is 4.4.8

MAUTIC – RESET BOUNCED EMAIL FLAG

Mautic has a feature where not only is a bounced email added to the bounced-emails segment, it also has a Bounced Flag, that has to be reset for you to email a user again. You can remove it one by one for each user, however if you have hundreds of thousands of users, you will need a more efficient way.

Here is that way

Log into your Mautic Server

Log into your Webmin Server

Click Servers

Click MariaDB Database Server

Select the Database

Click the Execute SQL button at the bottom

Execute:
DELETE FROM lead_donotcontact WHERE reason = 2

The flag will be reset

Back in Mautic..

Go into Contacts

In the Filter box enter in:
segment:bounced-emails

Press Enter

Select All contacts and remove them from the segment (DO NOT DELETE THEM)

and you can now email them again….

HOW TO UPDATE PI-HOLE

Quick How-To for updating pihole on Ubuntu

Open a terminal

Type:
sudo apt-get update
Enter in your password

Type
sudo apt-get dist-upgrade
Enter in your password

Type
pihole -up
Enter in your password

Pihole updates

That is it!

EMAIL WARMING USING POWER AUTOMATE – SENDING TO DOMAIN

For those of you who need to get a positive reputation on your domain for email campaigns, you first need to warm up the domain. Here is a simple power automate flow to assist with the task.

Login into Office.com

Click on the Menu to the Top Left of the screen

Select Power Automate

Click My Flows

Click +New Flow

Select Scheduled Cloud Flow

Give it a Name, Start Date, and Interval on how often you want it to run

Click Next

Click + Next Step

Type in mail and press Enter

Select Send an Email V2

Put in the address of a user for the new domain you want to send email to
ex: user@tacticalware.com

Enter in a Subject

Add some Text to the Body

Click Save

Test your Flow, and email should go though now.

Next you will want to setup an auto reply on the receiving email address, and also setup a power automate flow from the email domain so you can generate content coming out of it.

GODADDY – BACKUP DNS ZONE

Here is how to backup your DNS information in GoDaddy

Login into your account

Navigate to https://dcc.godaddy.com/control/portfolio

Identify the Domain name you want to backup

Click the three dots next to it “…“

Select Edit DNS

On the DNS MANAGEMENT page, next to the ADD button, click the Three Dots “…“

Select Export Zone File

Your DNS will backup to your computer.

Save it to somewhere special, and go on your merry way

INSTALL NVIDIA DRIVERS, CUDA, TENSORFLOW AND ANACONDA ON UBUNTU 20.04

Quick note on how to install the packages. It is in no particular order, these are raw notes my from my temporary build

Install Ubuntu

Open a terminal
sudo bash
apt-get update
apt-get dist-upgrade
nvidia-smi


Pick the version you want to use
apt install nvidia-utils-510
reboot


Open a terminal
sudo bash
nvidia-smi
ubuntu-drivers autoinstall
apt-get install curl


Go to the Anaconda site and download the latest build. For me it was the one below
cd Downloads/
bash Anaconda3-2021.11-Linux-x86_64.sh
source ~/.bashrc
conda info
conda update conda
conda update anaconda
sudo apt install nvidia-cuda-toolkit
reboot


Open a terminal
sudo bash
anaconda-navigator


Open a new notebook
pip install –upgrade tensorflow
Press run


Open a new notebook
import tensorflow as tf
print(tf.version)


Tensorflow will show the version

SCRAPE HISTORICAL FINRA SHORT DATA

To quickly get all the data from this page and all the sub pages:

You will need to run the following on a UBUNTU box

Open a terminal

mkdir finra-historical && mkdir all-years && mkdir downloads && cd finra-historical/downloads

wget -r -np -c -H https://www.finra.org/filing-reporting/trf/trf-regulation-sho-2020

after the scrape completes type

mv regsho.finra.org ../allyears

cd ../

rm -R downloads

This will move only the downloads you want, over to the directory called all-years, and it will delete all the data that is extra

SCRAPE DAILY SHORT DATA FROM FINRA

Here is a quick how-to on getting the daily short data. From this I look to see if the short volume has increased or decreased on a particular stock by leveraging a Power BI Dashboard that I had created. This guide is to show the simple command I run to initially scrape all the data

From a Linux box

Open a terminal

Type:
mkdir FINRA
cd FINRA
wget -r -np http://regsho.finra.org/regsho-Index.html

After it downloads the data I run
rm *.html

To remove the additional files that are not relevant, and then I move the data to where I need it

EXPORT TELEGRAM CHAT HISTORY FOR DATA ANALYTICS

From time to time you come across very valuable telegram channels with great information in them. You can run analytics on the chats, to identify trends and make decisions based on the data. To export the data perform the following:

Download and install the Telegram Desktop App
Login using your phone number and the code that Telegram sends you in the Phone App
Click on the Telegram Channel that you want to export
Click on the three dots at the top right of the chat window
Select Export Chat History
Place a check mark in all of the boxes, set the size limit to 2000MB and then click export
This will give you all the data.
If you want chats only, make sure you do not place checkmarks in all the extra fields
Click Export
Click Allow from your phone (yes you need to use both the desktop app and your phone app)
After 24 hours or so the Data will be available for you to download