SETUP ITT BACKEND

server requirements

• uvicorn
• fastAPI
• pydantic
• apscheduler
• python-multipart
• python-dotenv

worker requiremets

• pydantic
• httpx
• python-dotenv

frontend requirements

• node >= 22
• npm >= 10.14.1

install deps

pip install fastapi pydantic uvicorn httpx apscheduler python-multipart python-dotenv cd frontend; npm i

start local backend server

cd backend; server.sh

start worker

cd backend; worker.sh

start frontend

cd frontend; npm run dev

---

Docker Quickstart

[!NOTE]

Please run the docker compose commands from the root of the project

Build & Run

docker compose up --build -d

Build only

docker compose build

Run only

docker compose up

Stop & Clean Up

# Stop containers
docker compose down

# Stop and delete built images
docker compose down --rmi all

# Stop and delete shared data volume
docker compose down -v

---

Bedienungsanleitung

Um einen download zu starten, muss zunächst eine Konfiguration anelegt werden. Als beispielparameter kann ihr zum Beispiel "-x --audio-format best --no-keep-video --embed-thumbnail --convert-thumbnails png --embed-metadata --embed-chapters -4" gegeben werden. Sie benötigt nur noch einen NAmen, der REst kann auf den Standardwerten gelassen werden.

Nun muss ein Profil angelegt werden. Die Konfiguration wird diesem automatisch zugeordnet, da es nur eine gibt. Nun braucht es nu noch einen Namen, der Rest kann auf den Standardwerten gelassen werden.

Nachdem das Profil gespeichert wurde kann man diesem Links hinzufügen. Durch "Run Now" wird das Profil dann ausgeführt und man kann den Fortschritt auf der Statusseite beobachten.

TODO

backend :

• optimize patch queries
• handle empty profile / config names # ignored
• protecting cookies from the cookie monster # wont do
• maybe ditch the cleanup type and instead have 2 cleanup_param fields so that both storage-based and time-based rules can apply simultaneously # OOS
• maybe hold on to archive entry filename after deletion # maybe
• stream the files during download to avoid overflowing the tmpfs, possibly impossible
• worker ID, tracking workers in backend
• use cookies in discovery process

frontend :

• remove any whack isNaN and generalize id param in routes
• be careful of pagination in selects/searchable selects
• store column selected state in local storage
• global option (probably under misc page) for default page size
• make source links searchable and editable

DONE

• FE: dark mode
• FE: multi selct profile links in profile page & delete all source links for profile
• FE: jobequeue: resolve manually jobs
• FE: add support for source link names
• Workers
• Console Feed
• API
• search in paginated requests
• use tempfile in worker for cookie
• forbid certain arguments
• endpoint for updating
• allow patching of cleanup_priority in content_link
• added name-field to content_link class referencing name of parent profile
• allow multiple feed clients
• transfer files over http
• add optional name-field to source-links
• keep currently being worked on jobs in memory
• add update event for changes in frontend
• discovery and scraper queue are now get-able, with pagination
• added name-field to job which contains source-link / profile name, depending on job type
• figure out direct content links in discovery
• proper state-machine implementation for content_link
• send stderr to feed if download fails
• move deletion to backend
• autodelete
• worker interrupt handling
• worker : download files to tmp
• keyboard interrupt handling
• cleanup after failed update
• server path prefix
• .env
• file existance testing in autodelete / watchdog

DISCARDED

• move filesystem stats out from db to dynamic task for worker, use asyncio future
• deletion in workers

NOTES

• Maybe re-add size to content_link to enable sorting by file size