Dave

#api After upgrading php-fpm on the api front-end servers this morning I just noticed the fpm default config was in place so the api is running like a dawg. Restoring the pre-tuned settings now...

#api Podcast index api servers are patched and serving data again.

RE: https://podcastindex.social/@steven/117475936422223299

#api Heads up, if Podcast Index api goes down at 9:00 am central time you'll know why. Will know more when the announcement hits.

#api Just pushed a new change that will hopefully auto-merge all of those duplicate wordpress podcast feeds across the Index. So far it looks promising. It will take a few days to fully sweep the whole Index.

#api I fixed a huge problem with the feed aggregators. Based on this snapshot of our database metrics, can you guess at roughly what time that happened?

#api There was a problem with search ranking in the search/byterm endpoint after switching the db schema. This should be fixed now. Will explain in the board meeting tomorrow.

#api All out brawl with the feed aggregators this morning. I'm about to win though.

#api Adding 2 new indexes on the feeds table. Expect the api to choke for a few minutes until this completes.

#api All api endpoints are now live with the new database schema. Bug reports welcome. I'm tired. 😅

#api The schema refactored episodes/byfeedid, bypodcastguid, byitunesid, byfeedurl and value/bypodcastguid endpoints are now live.

#api The new podcasts/byitunesid endpoint is live.

#api The podcasts/byguid and podcasts/batch/byguid endpoints are now flipped to the new db schema.

#api First run of the itunes ID 404 reconciler under the new podcast identity regime. 🤞

#api The new podcasts/byfeedid endpoint is deployed. That's all for today. Will watch the logs until tomorrow.

#api The podcasts/byfeedurl endpoint is deployed with the first changes to the new podcast identity layer. We will do these one at a time, watching for problems.

You will see nothing different in the response shape under normal circumstances. The response "feed" object's id property now means the podcast ID. The feedId property now means the underlying feeds table row that podcast is treating as canonical.

#api Making some final refinements to the new api endpoints before deploying. These will be the first ones using the new podcast identity layer. 😰

#api I'm running a script to sync up podcasts.id and newsfeeds.id so that when a new feed is born they are in sync. During the cutover to podcasts.id as the canonical form, there were about 11k feeds that drifted away from being in sync with their podcast because of a gap in the auto_increment values.

This isn't strictly necessary, it just will make human review a little less cumbersome if those values track everywhere possible.

#api Refactored the weekly database dump script since it was getting slower and slower and was very old. Re-running the new one now to test. I'm anticipating 4 hour run instead of 28 hours.

#api All feed ingestion paths now use the new 'podcasts' identity table. Spent the morning cleaning up a lot of old error log debris I've been neglecting for years so I can have clean logs for the next step, which is feed <-> podcast reconciliation (the whole point of this project).

No API endpoints read from the podcasts table yet. That comes later. This phase of work is making sure the podcasts table is the proper point of authority over the feeds table.

#api I would like to pretend (by silence) that I didn't break feed ingestion this morning for about 20 minutes by forgetting to update the env files on the aggregator servers. But, I'm sure

will produce some sort of graph showing a dip this morning and asking what's up. So, I'm proactively admitting it.

#api The new podcasts table is now being written to by default on all new incoming feeds and the conflict queue is being populated for merge decisions. This is step 1 of the scary stuff. Let's hope nothing collapses.

#api Podcast Index database refactor is progressing. Each step is a baby step until the last one, which is a big leap all at once into the new schema across all endpoints and utility scripts. To say I'm nervous would be an understatement.

#api Currently back-filling the new 'podcasts' table in the database with the proper PI feed ID's. Api might run slow for the new little while.

#api Linode emergency maintenance on our db host. API is down.

#api I'm in the middle of a big refactor with the Podcast Index database schema. If I do my job right, nobody will notice it. But, the side effect will be an overall huge increase in the consistency of the data (duplicates, url change lag, etc).

#api The new episodes by person endpoint:

/api/1.0/episodes/byperson?q=James+Cridland&max=5

It reports directly what the level podcast:person tags dictate.