Skip to content

Commit

Permalink
Add Poll Tweets functionality
Browse files Browse the repository at this point in the history
See README changes for a description of the new functionality. The implementation is at the "get it working stage", not at the "get it right" stage. I'm also not a Typescript developer and have definitely committed some code violations. But it's powerful functionality and I'd like to get it out there early.

Known issues:
  - If settings like "noteLocation" or "pollFilename" are empty, strange things will happen. Defaults are not enforced.
  - An error like "images failed to download" sometimes appears. But images seem to be downloading. More investigation necessary.
  - There is very little feedback on progress to the user. This is challenging to do well.
  - The process cannot be interrupted (or monitored, easily), so mistakes can result in a lot of downloaded tweets!
  - Every Tweet duplicates a lot of front matter. On reflection, this is still better than any alternative I could think of.
  - Speaking of, the "fetch new tweets" functionality relies on frontmatter being present and will quietly download all tweets again if it is not there.
  - There is now a lot of code repetition. This is so my additions are well contained, rather than modifying too much existing code at this stage.
  - Testing, and tests, are basically non-existent at this stage.
  • Loading branch information
hraftery committed Nov 19, 2022
1 parent c296742 commit c6a9e2d
Show file tree
Hide file tree
Showing 7 changed files with 474 additions and 1 deletion.
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
## About This Fork

This project is a small fork of the excellent [Obsidian Tweet to Markdown](https://github.com/kbravh/obsidian-tweet-to-markdown) project. This fork adds **Poll Tweets** functionality to automatically fetch new tweets from a Twitter timeline and add them to your Obsidian vault.

The feature can be found at bottom of the existing plugin settings, under the "Additional Settings for Polling Tweets" heading. Enabling polling will periodically check for new tweets from the Twitter handles specified, and prepend them in reverse chronological order (just like appears in the normal Twitter user timeline) to a Markdown file named using the pattern specified. If the file does not exist, it will be created and *all* tweets added (up to the API limit which is 3200 at the time of writing).

<p align="center">
<img src="images/poll-tweets-screenshot.png" alt="Logo" height=250>
</p>

**Note:** the current implementation is alpha "just-working" stage. It may not work with settings other than the defaults, errors may not be reported effectively and the code is rough. Improvements welcome!

The README from the original project appears below. Please do not bug the original author with issues associated with the **Poll Tweets** feature in this fork.

---

## Original Readme


<br />
<p align="center">
<a href="https://github.com/kbravh/obsidian-tweet-to-markdown">
Expand Down
Binary file added images/poll-tweets-screenshot.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
24 changes: 24 additions & 0 deletions main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import {pasteTweet} from 'src/util'
import {Tweet} from 'src/types/tweet'
import {TweetCompleteModal} from 'src/TweetCompleteModal'
import {TweetUrlModal} from 'src/TweetUrlModal'
import {createPollManager, PollManager} from 'src/pollManager'

interface PasteFunction {
(this: Plugin, ev: ClipboardEvent): void
Expand All @@ -16,6 +17,8 @@ export default class TTM extends Plugin {
bearerToken: string
tweetComplete: TweetCompleteModal
pasteFunction: PasteFunction
pollManager: PollManager
pollIntervalID: number //should be ReturnType<typeof window.setInterval> but that seems to return the wrong type

async onload(): Promise<void> {
console.info('Loading Tweet to Markdown')
Expand Down Expand Up @@ -71,6 +74,8 @@ export default class TTM extends Plugin {
})

this.addSettingTab(new TTMSettingTab(this.app, this))

this.pollManager = createPollManager(this.app, this)
}

onunload(): void {
Expand All @@ -83,5 +88,24 @@ export default class TTM extends Plugin {

async saveSettings(): Promise<void> {
await this.saveData(this.settings)

if(this.settings.pollEnabled) {
if(this.pollIntervalID)
clearInterval(this.pollIntervalID) //registerInterval will still try to cancel it when unloading, but so be it.

this.pollManager.pollRun();

var numHours = 24 //by default
switch(this.settings.pollFrequency) {
case 'monthly': { numHours = 24*30 }
case 'weekly': { numHours = 24*7 }
case 'twoDaily': { numHours = 24*2 }
case 'daily': { numHours = 24*1 }
case 'twiceDaily': { numHours = 24/2 }
case 'hourly': { numHours = 1 }
}
this.pollIntervalID = window.setInterval(() => this.pollManager.pollRun(), 1000*60*60*numHours)
this.registerInterval(this.pollIntervalID)
}
}
}
150 changes: 150 additions & 0 deletions src/pollManager.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
import {App, Notice, TFile, parseFrontMatterEntry} from 'obsidian'
import TTM from 'main'
import {createDownloadManager, DownloadManager} from './downloadManager'
import {buildMarkdown, getBearerToken, getUser, getTimeline} from './util'
import {createFilename, doesFileExist, sanitizeFilename} from './util'
import type {Tweet, User} from './types/tweet'
import { TweetCompleteActions } from './types/plugin'


export const createPollManager = (app: App, plugin: TTM): PollManager => {
const pollRun = (): void => {
if (!navigator.onLine) {
new Notice('Polling for new tweets skipped because you appear to be offline.')
return
}

const bearerToken = getBearerToken(plugin)
if (!bearerToken) {
new Notice('Polling for new tweets skipped because bearer token was not found.')
return
}

const handles:string[] = plugin.settings.pollHandles.replace(/\s|@/g, '').split(",")
var downloadManager = createDownloadManager()
plugin.bearerToken = bearerToken

handles.forEach(async handle => {

let user
try {
//Note the term for a "handle" in the Twitter API is "username"
user = await getUser(handle, bearerToken);
} catch (error) {
new Notice(error.message)
return
}

//Hella hacky way to make use of the existing createFilename interface
const dummyTweet:Tweet = {
includes: {users: [user]},
data: {id: "", text: "", created_at: "", author_id: "",
public_metrics: {retweet_count: 0, reply_count: 0, like_count: 0, quote_count: 0}}
}

let fname = createFilename(
dummyTweet,
plugin.settings.pollFilename,
{ locale: plugin.settings.dateLocale, format: plugin.settings.dateFormat })

const fpath = createFilename(
dummyTweet,
plugin.settings.noteLocation ? plugin.settings.noteLocation : "./", //fix broken location default
{ locale: plugin.settings.dateLocale, format: plugin.settings.dateFormat },
'directory')

if (!fname || !fpath) {
new Notice('Failed to create filepath for timeline.')
return
}

if (! await app.vault.adapter.exists(fpath)) {
await app.vault.createFolder(fpath).catch(error => {
new Notice('Error creating tweet directory.')
console.error(
'There was an error creating the tweet directory.',
error
)
return
})
}

//Seems getAbstractFileByPath doesn't handle "./" path. Strip it out if necessary here.
let abstractFile = app.vault.getAbstractFileByPath(fpath == "./" ? fname : `${fpath}/${fname}`)
let file: TFile
let since: Date
if(abstractFile && abstractFile instanceof TFile) {
file = abstractFile
if(file)
{
let metadata = app.metadataCache.getFileCache(file)
//If the parse fails, date will be the unix epoch, which is as good as undefined.
since = new Date(parseFrontMatterEntry(metadata.frontmatter, "fetched"))
}
}

let tweets: Tweet[]
new Notice(`Polling for new Tweets from ${handle}...`)

try {
tweets = await getTimeline(user.id, since, bearerToken)
} catch (error) {
new Notice(error.message)
return
}

plugin.tweetMarkdown = ''

const markdowns = await Promise.all(
tweets.map(async tweet => {
let markdown
try {
markdown = await buildMarkdown(app, plugin, downloadManager, tweet, 'normal', null)
} catch (error) {
new Notice('There was a problem processing the downloaded tweet')
console.error(error)
}
return markdown
})
)

if(markdowns && markdowns.length > 0) {
plugin.tweetMarkdown = markdowns.join('\n\n---\n\n')

await downloadManager
.finishDownloads()
.catch(error => {
new Notice('There was an error downloading the images.')
console.error(error)
})

// clean up excessive newlines
plugin.tweetMarkdown = plugin.tweetMarkdown.replace(/\n{2,}/g, '\n\n')

// write the note to file
if(file)
{
var t = await app.vault.read(file)
app.vault.modify(file, [plugin.tweetMarkdown, t].join('\n\n---\n\n'))
new Notice(`${fname} updated.`)
}
else
{
const newFile = await app.vault.create(`${fpath}/${fname}`, plugin.tweetMarkdown)
new Notice(`${fname} created.`)
}
}
})

// clean up
plugin.tweetMarkdown = ''
}

return {
pollRun,
} as PollManager
}

export interface PollManager {
pollRun: () => void
}
84 changes: 83 additions & 1 deletion src/settings.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import {App, moment, Platform, PluginSettingTab, Setting} from 'obsidian'
import {App, moment, Platform, PluginSettingTab, Setting, ToggleComponent} from 'obsidian'
import {TweetCompleteAction, TweetCompleteActions} from './types/plugin'
import {FolderSuggest} from './suggester/folderSuggester'
import TTM from 'main'
import { exists } from 'fs'

export interface TTMSettings {
bearerToken: string | null
Expand Down Expand Up @@ -29,6 +30,10 @@ export interface TTMSettings {
includeDate: boolean
dateFormat: string
dateLocale: string
pollEnabled: boolean
pollHandles: string
pollFilename: string
pollFrequency: 'monthly' | 'weekly' | 'twoDaily' | 'daily' | 'twiceDaily' | 'hourly'
}

export const DEFAULT_SETTINGS: TTMSettings = {
Expand Down Expand Up @@ -56,13 +61,18 @@ export const DEFAULT_SETTINGS: TTMSettings = {
includeDate: true,
dateFormat: 'LLL',
dateLocale: 'en',
pollEnabled: false,
pollHandles: '',
pollFilename: 'Twitter timeline - [[handle]]', //Unfortunately, seems to displace the placeholder but how else do you set a default?
pollFrequency: 'daily'
}

export class TTMSettingTab extends PluginSettingTab {
plugin: TTM
locales = moment
.locales()
.reduce((obj, locale) => ({...obj, [locale]: locale}), {})
pollEnableToggle: ToggleComponent

constructor(app: App, plugin: TTM) {
super(app, plugin)
Expand Down Expand Up @@ -405,5 +415,77 @@ export class TTMSettingTab extends PluginSettingTab {
await this.plugin.saveSettings()
})
)

containerEl.createEl('h2', {text: 'Additional Settings for Polling Tweets'})

new Setting(containerEl)
.setName('Polling enabled')
.setDesc(
'Periodically poll for new tweets.'
)
.addToggle(toggle => {
toggle
.setValue(this.plugin.settings.pollEnabled)
.onChange(async value => {
this.plugin.settings.pollEnabled = value
await this.plugin.saveSettings()
})
this.pollEnableToggle = toggle
})

new Setting(containerEl)
.setName('Twitter handles to poll')
.setDesc(
'Handles (with or without @ prefix) to poll for new tweets, separated by commas.'
)
.addText(text =>
text
.setPlaceholder('@handle1,@handle2')
.setValue(this.plugin.settings.pollHandles)
.onChange(async value => {
this.plugin.settings.pollHandles = value
this.pollEnableToggle.setValue(false)
await this.plugin.saveSettings()
})
)

new Setting(containerEl)
.setName('Filename to update with new tweets')
.setDesc(
'The name of the files to add new tweets to. Files are created if they don\'t exist. You can use the placeholders [[handle]], [[name]], [[text]] and [[id]]. Defaults to "Twitter timeline - [[handle]]"'
)
.addText(text =>
text
.setPlaceholder('Twitter timeline - [[handle]]')
.setValue(this.plugin.settings.pollFilename)
.onChange(async value => {
this.plugin.settings.pollFilename = value
this.pollEnableToggle.setValue(false)
await this.plugin.saveSettings()
})
)

new Setting(containerEl)
.setName('Poll frequency')
.setDesc(
'How often to poll for new tweets.'
)
.addDropdown(dropdown =>
dropdown
.addOptions({
monthly: 'Monthly',
weekly: 'Weekly',
twoDaily: 'Every second day',
daily: 'Every day',
twiceDaily: 'Twice daily',
hourly: 'Hourly'
})
.setValue(this.plugin.settings.pollFrequency)
.onChange(async (value: 'monthly' | 'weekly' | 'twoDaily' | 'daily' | 'twiceDaily' | 'hourly' ) => {
this.plugin.settings.pollFrequency = value
this.pollEnableToggle.setValue(false)
await this.plugin.saveSettings()
})
)
}
}
25 changes: 25 additions & 0 deletions src/types/tweet.ts
Original file line number Diff line number Diff line change
Expand Up @@ -119,3 +119,28 @@ export interface User {
username: string
profile_image_url: string
}

export interface UserResponse {
data: User
errors?: Error[]
// other error fields
reason?: string
status?: number
}

export interface Meta {
result_count: number
newest_id: string
oldest_id: string
next_token?: string
}

export interface Tweets {
includes: Includes
data: Data[]
meta: Meta
errors?: Error[]
// other error fields
reason?: string
status?: number
}
Loading

0 comments on commit c6a9e2d

Please sign in to comment.