we're switching to an idiom where the bot is only on one connection, so
we don't need to care about connection tracking. new_sendmsg accordingly
doesn't take a connection argument. now i can remove the old sendmsg
caught in the wake, a bunch of connections being passed here and there
can be removed, changing some module method signatures and such. there
might be more to remove still
a bunch of logic is moved around, some queries are improved, max_size
does what it's actually supposed to do. all in all this is a much
clearer chainer, even if the actual results are more or less the same.
it's probably a bit faster in most cases but slower in situations when
all the seed words have been consumed and it needs to do
__start1,__start2 chains (since there's so many of them, it's rather
slow). otherwise, it tries to use seed words in sentences, combining
multiple sentences when possible. there's a lot more in the periphery,
but that's the general idea
whether or not we went backwards and forwards, or just forwards, this
cycle of the loop, end the iteration by calling the end of the sentence
our hit word. if it was our seed word, this will trigger a new seed
selection
get one random v for a k1,k2 via SQL. prefer a word to show up in the
results, though there's no guarantee it will. this simplifies the
general looking forward case, and could possibly even work ok on the new
sentence stuff, though i haven't tried to update that portion of the
code yet
only add the reverse-search result to list of words if it isn't __start2
(and if it is __start2, just carry on, giving the code one last chance
to find something else)
rather than getting all k2s for a value from the database, then walking
the list and picking one at random, pick one for a value at random via
a query
this simplifies the code, and is (usually) faster than the old way,
which has been removed. it would be even faster if it weren't for that
context_id stuff, but so it goes
the code, in a kind of trial state, would very quickly stop trying to
work backwards. (part of this was for performance reasons, i believe.)
since that seems to have proven stable, let's mess with it --- the
backwards chainer can now go backwards a random distance, rather than
just what almost always turned out to be 2
another "this is unnecessary" change, obviously impacting all the
modules that override __init__ as well as the base class. again, they
can use the DrBotIRC instance for anything, which is (with one
exception) only for add/remove_global_handler, which i'm planning on
working my way off of anyway
obviously this means all of the modules changed to accomodate. this is
one of many steps to reduce the number of times we pass connections and
servers and other such info around, when it's mostly unnecessary because
modules have a reference to DrBotIRC
when starting another sentence because the main one is too short,
do a bit of work in an attempt to avoid "nick: blah" starts, since
they're fairly common. instead we just ignore nick: and start with
"blah blah"
when starting a second (or Nth) chain because the results so far
are too short, add punctuation to the end of the chain, just to
make things feel a bit more natural
this clarifies a bunch of sections and seems slightly faster
target_word (which would be randomly selected from the input every
time) is replaced with seed_words, a shuffled list from the input.
this is to eliminate accidental reuse of the target word, which
would result in chains like X X X X X X X X X X X X X because
it'd keep targeting X
the rest of this is mostly just debug cleanup, though to simplify
the backwards code it only tries to find one target word
WARNING!
there's no going back now. this change is *huge* but it was overdue.
WARNING!
the database backend is now mysql. modules that should use a database
but don't yet were left untouched, they'll come later. scripts haven't
been converted yet, though i'm pretty sure i'll need to soon.
while i was going through everything, connection/cursor idioms were
cleaned up, as were a bunch of log messages and exception handling. this
change is so gross i'm happy things appear to be working, which is
the case --- all modules are lightly tested.
for simplicity's sake, this was added to the extlib/irclib rather
than subclassing. because i'm lazy. anyway, check that flag instead
of doing the event._target = None hack, since that hack was breaking
Markov.
for an unrelated reason (what to learn and not learn), update Markov
also remove an unused method that was getting in my way while coding this
this should result in no chains having a null context --- if no pre-existing
context is created, one is created for the channel/nick and used. this makes,
for example, arbitrary queries "private" to that nick (again unless that has
been overridden). shouldn't affect much of anything, but adding this made
the context-less learning code obsolete, which is fine since it was never used
anyway
the module will drop your old tables if you have them, so if there's data there,
be sure to back them up and figure out some migration strategy (probably annoying
and probably having to script it).
the big change is that each line is associated to a context now, and channels
are also associated to contexts. this should allow for a better partitioning
of multiple brains, and changing which channels point to which brain.
also caught in the wake is some additional logging verbosity, and a change to
no longer lower() everything learned.
the script to dump a file into the database has also been updated with the above
changes
this only makes sense if we have a target word set, which we usually do.
start with the target word and go backwords, finding k2s that lead to it
(and that lead to that k2, and so on) until we get to the start-of-chain
value, when we know we're done working backwards. then resume the normal
appending logic
probably needs some work, probably a bit slow on huge databases. analysis
pending, but this appears to work
check interval is every 10 minutes, rows in markov_chatter_target
have a 1 in chance chance of leading to a line being generated,
every 10 minutes. (so an interval of 144 = 10 min * 6 * 24 = one line
per day, on average)
Markov, Twitter: switch to forking a thread ourselves, and check every
second whether or not to quit. this is the "better" part above, as
now we can instantly quit the thread rather than waiting for all
the timers to fire and expire
Brian S. Stephan