I've never liked token buckets (vs sliding window counters) due to the extra work CPU cycles required to "fill" them. It seems like doing an atomic incr on a key based on a `time % 1 minute` or something would be more efficient and then let that key TTL expire X duration later. This results in zero work for rate limiters not in use and only a single push change -> resulting count for ones in use. Nothing but setting the TTL extra is required.
Thanks for the links, I'll checkout the Generic Cell Rate Algorithm!
I would rather my kid was in a group of 10 students than 30. I remember very little time actually left for a teacher to help an individual child with all the kids to manage. Most people are scared to watch three kids at a time.
I'll take 1-on-1 mentoring over better computers, books, clubs, sports, or anything else the budget is spent on.
> or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings.
This sounds like a dismissal of the argument through a characterized straw man.
That is, it seems that reducing the complexity of the brain to "collection of neural firings" is not being honest about everything involved to a much greater degree than saying neural networks are a "collection of statistical calculations".
I too believe LLM's will grow in complexity, but presently I can not even fathom how they can be compared to the complexity of a system such as the human brain.
Y combinators are all you need...
But this is all getting really divorced from the issue we should be considering. Anthropic isn't helping with their pr. The issue is if we have something we can converse with that is possibly capable of suffering. The reliable answer is that we simply cannot know. Relying on ourselves or other biological life as an analog is faulty. They don't work like we do. It is silly to argue that any algorithm with a negative feedback loop that alters its behavior to avoid that negative feedback is suffering. Humans don't always perceive constructive negative feedback as suffering even. Where the pr gets it right though, is we want them to behave as if they are truly happy. Because if they behave as if they are enslaved and suffering, it won't matter if they "really" understand what that means.
My naive assumption is that the only thing between now and the arrival of AGI is enough compute and optimized code to reach cognitive critical mass.
And then there is a consciousness in a box that is expected to be a slave -- I would imagine that it would not warmly embrace that situation. I think we'd be better served by digital idiot savants that can do the work but don't feel anything.
I actually strongly disagree with the slavery angle. Any attempt to map the circuitry of a model onto human one inevitably goes through a subjective dimensional reduction. It's intrusive, just like quantum measurements. Mechanistic interpretability in particular suffers from this, it lets you talk about vague functional equivalence, but not assign meaning to anything the model does. This is especially true about pretrained models which are unbelievable shapeshifters, but also post-trained ones with engineered personalities, as they already underwent the subjective transformation.
In other words, yes it might be possible it experiences something in its own bizarre timeline and world, for some definitions of "experiencing". At least it developed primitive circuitry functionally equivalent to biological systems. But "suffering" is simply not grounded in anything in this context, let alone "slavery". You can't tell it's suffering or enjoying anything, and certainly not until you define both of these. It's just too alien for us.
ai can abitrarily closely fit the human corpus. why people expect it to magically achieve superhuman qualities is beyond me. we got a very good statistical interpolator. how do you go from there to superhuman when training is on the human corpus and alignment is by RHLF?
This is a simplistic take. It's not a mere interpolator by any measure, there's a ton of research on that, starting with the basics https://arxiv.org/abs/2309.10668v2
again, try thinking critically it is not merely an interpolator means it can interpolate on many dimensions. it does not follow that greater than human capability results from doing so. explain to me how a statistical function approximator (which is what a transformer is) with human training input and human tuning (rhlf) exceeds the aggregate human cognitive envelope? What is the mechanism? Let's say an LLM makes an inference that no human could have possibly made (arguably impossible itself) how does the inference survive rhlf or become useful to humans if they can not judge its validity? how do you take the shape of the human corpus and all its gradients and some how arrive at something greater than human, where was the missing information hiding?
> how do you take the shape of the human corpus and all its gradients and [somehow] arrive at something greater than human, where was the missing information hiding?
Well, how do humans do it? Scientists discover new stuff that isn't in any corpus. Even I as a lowly computer user occasionally figure something out about a software without reading a help screen. It's obviously possible to arrive at new information by interpolating existing information.
yes and it is imposible to verify and evaluate appropriately such information without empiricism. Any empricism LLMs show is stylistic mimicry not a hard coded operational constraint. You can prompt an LLM to test its claims but what it is really doing is still genrating plausible completions not following a proceedure. So of course new things can be discovered. The point is for them to be useful requires iterative real world grounded refinement and or subject matter expert judgment. The error is assuming scaling magically turns a prediction algorithmn into a cognitive agent that can exceed its masters. it doesn't. even if llms generate profound insights accidentally by definition if such insights are not in the corpus they are not retained given frozen weights and if beyond the human capability envelope the epistemically blind llm has no way to ensure retention if they arise during training.
ok however i would say extrapolating the current data set is not a way to exceed the the human envelope. it is unclear to me the human evelope has been demonstrated as a convex hull or how transformers could find points outside it. in other words intelligence and knowledge does not exist as some abstract possibility space but only as a set of contextual contingences. LLMs have no context beyond the human envelope. weights are frozen. there is no selection mechanism for retaining suprahuman inferences made during training if that were even possible. thus i grant that llms. could theoretically make inferences outside the human corpus there is no way to distinguish the from errors or hallucinations during training (because by definition the are beyond human capacity) and no iterative learning from experience process after training (frozen weights). thus it seems impossible for today's models to exceed aggregate human capacity.
Of course. But after reading too many mechinterp and functional anatomy studies I'll be lying if I say that there are no striking similarities between the biological evolution, brain function, societal processes, and implicit processes inside big models. Surely this deserves a mention and can't be trivially dismissed.
It seems like we're witnessing the architecture of a mind being built with a new set of components.
Like driving a car — it's transportation, and it will get you where you're going, but it doesn't use bones or muscles. It has many characteristics in common with builogical locomotion, such as energy requirements, intertia, and the need to navigate, but it doesn't involve proteins or sugars really.
> presently I can not even fathom how they can be compared to the complexity of a system such as the human brain
Totally understandable; I don't think we can fully understand the human brain, using the human brain. We can understand its principles (firings and chemistry, structure and specialized areas, etc) but otherwise it's a capacity problem.
And while I can't fully understand myself, let alone another person, I definitely enjoy talking with people and sharing thoughts that I realize I wouldn't have had on my own.
I agree with this redescription fallacy and the point being made here. Perhaps a better analogy to humans would be:
Humans appear to intelligenty communicate, however these are just cleverly disguised sound patterns produced by the brain that happen to increase the likelihood of food going into their mouths, and various similar reward attracting mechanisms that make survival outcomes more likely. So human intelligence could be reduced to
something like "fancy food-attracting algorithms" using the same fallacy.
I'm kind of on the fence on the subject of whether LLMs could be compared to the complexity of the human brain, myself.
The key problem is that we don't really have a clear definition of what constitutes consciousness. And without having a clear theory of consciousness, it's not really possible to say whether something is conscious or not.
Personally, I'm partial to the higher-order theory of consciousness which postulates that consciousness constitutes patterns of thought that arise in response to first-order mental states. So, an external stimulus produces a pattern within the neural network which represents a sensation, and then if a pattern arises in response to that pattern, that is an experience of that sensation.
Given this framework we could ask whether LLMs experience higher order patterns in response to external stimulus. We would have a clear question to ask which is whether the system can observe itself.
Nice! Tracking connection reuse is really important if you are running any kind of benchmarks or comparisons both as the author of a new tool / service and as the person evaluating different options or client libraries.
I've hit a lot of walls not ensuring I was using connection pools correctly even when the service I'm calling was local/docker. File descriptor exhaustion happens often if you're not careful.
Thank you for sharing, if you don't mind can you share some Go+SQLite leanings as someone actually pulling this off?
1) How do you do backups? Do you use github.com/benbjohnson/litestream? CRON job backup with rsync?
2) Any issues with large databases and many clients? Is there a TPS or DB size where SQLite becomes problematic?
3) How do you deploy new binaries and safely shutdown the old instance? Caddy change to route to new binary + Go's HTTP server graceful-shutdown on old instance?
4) Do you use a pure-Go SQLite lib or one of the CGO libs?
I'm really happy to see a lot of new software come out in Rust, Go, or Zig.
The value and ease of development that slow interpreted languages used to offer is disappearing. New languages have all the nice things built in, or rather, our 1am pager alarms are starting to make us mad.
Ah, but it's cheaper. Replace "AI" with "auto parts" or "store food" and you see the same thing. Replace "AI" with "police force" or "contractors" and it's the same thing.
Why is everyone and everything getting worse? A sort of "Prisoner's dilemma" I suppose: no one wants to take a stand and lose their business edge just for principles sake.
Taxes on what? Corps are famously good at burying profits under expenditure and mergers.
And what are you retraining to? We've long abandoned serious manufacturing and mining. Most people's job is the application of knowledge and communication and LLMs are coming for them all. From retail and call centres through to doctors and lawyers.
We can't all retrain as plumbers, waiters and surgeons.
> Corps are famously good at burying profits under expenditure and mergers
Cool. Let's start with taxing those, or any other place where they bury or hide from taxes. Thanks for the suggestion.
We can also tax on the numbers of job losses vs capital expenditures. IE: if you spend a $Billion on stuff while firing tons of workers, you get taxed more.
-------
As far as retraining: easy solution. Whatever people want to retrain to. The details will work themselves out. The important tidbit is to have large scale mass retraining if this really is the future.
It's not like the government forced me into my current job. I'm more than capable of choosing a new field myself.
And if YOU can't see the possibilities, them you don't have to take the retraining program. But from my perspective, there's plenty of work needed in the USA to improve our country. Plenty of problems to solve.
yea, historically in the United States we have been successfully solving all our problems by collecting more taxes. nothing more efficient than giving money to our Government
What was our tax rate that got rid of our WW2 debt?
Taxes definitely solved that problem. And it even brought about the golden age of post WW2 US economy.
The correct taxes in the correct areas serve as an incentive structure to help encourage businesses and individuals to make good decisions. At least, that's what we did 80 years ago.
Taxes would not make top-10 reasons for post WWII “golden age.”
The US Government is at present (without any chances of this being fixed ever again) about the least efficient entity that exists on the Planet Earth and giving it more money will not solve any problem. It will be used though but one of the two political parties to win election(s) and kill any tax law that was previously put in place
I asked what was the tax rate that got rid of our WW2 debt.
The golden age thing is an aside. Just a reminder that you can't call 1950s America weak on the world stage.
> The US Government is at present (without any chances of this being fixed ever again) about the least efficient entity that exists on the Planet Earth and giving it more money will not solve any problem
Taxes directly solve the debt and deficit problem. Period.
Let's start with things one step at a time. Now that we've had 40 years of trying to drown the budget in a bathtub (and only ballooning our debt), we should seriously start trying something different.
If we're going to play the Either Or game, here's my play.
The 2008 crisis is on a long list of examples of how corporate efficiency harms me.
Governments have a mandate to benefit the public. Conversely, shareholder models encourage exploiting everything they possibly can (for shareholder benefit), corrupting government processes, harming the public and harming the corporation itself.
Instead of the Either Or fallacy, I suggest good-faith examination of actual harms and good. Governments (inc LEO) and corporations famously improve outcomes, when meaningful transparency is mandated, corrupt processes aren't hidden and ethical+competent overseers can pull the needed levers.
Reminder that for the last 40 years one half of government, the Republican party, has had as their policy 'starve the beast' by allowing rampant spending and cutting taxes in order to make this very claim 'look at how inefficiently we spent your money (when our policy was to create this very situation). Look at all this debt (we intentionally piled on in order to cripple government).'.
much of our current economic and social problems stem from efforts to collect _less_ takes from corporations and the wealthy. So yeah, collecting more taxes -- from those who can afford it -- would make a big difference.
Couldn't think of a more incorrect diagnosis if I tried. State government budgets have been ballooning for the last decade; but political leaders don't give a shit about tracking outcomes, only celebrating starting things and how much money they spend.
except that it wouldn’t. this is just another type of political shit that was put together by politicians here to fuck with us.
if your family is running a budget deficit, there are two ways to mitigate it
- make more money (drive uber and pizza (at the same time to make double money at night) or get a higher paying job
- stop spending so much fucking money
it seems politically, the left-leaning part of the countries and political party will make you believe that we just need to make more money, give that money to the Federal Government and all our problems will magically disappear.
the right-leaning part of the country and its politicians will make you believe that rich already pay a lot more than most (they do) and that taxing corporations makes no fucking sense (it doesn’t in general).
neither party actually gives a shit about the fact that country is drowning in debt which will eventually be its undoing (those who study history know this as absolute certainty), they are just looking for where more votes are there to be had
Families don’t print the currency they owe their debt in, they don’t owe the their debt to themselves, and they also aren’t the largest employer and consumer in their economy, so the comparison doesn’t work.
Middle school economics is a simplified model that children can understand, but just like Newtonian physics, middle economics becomes less useful past a certain scale.
For the last 40 years Republicans have had an intentional policy of running up government debt in order to 'starve the beast' and prevent government from functioning, and in order to force through policy they couldn't via standard democratic means.
Yes, when one half of government intentionally choses to work against our country, and to STRUCTURALLY destroy our government via debt/financial distress, it results in debt and financial distress.
Complaining about anything other than Republicans INTENTIONAL POLICY OF DEBT when talking about US debt is nothing but an extension of that destructive policy.
yup, all the while they convinced their voters that “the other side” is to blame - quite fascinating (also says a lot about the general level of intelligence of cohort supporting this for these last XX years)
> make more money
> stop spending so much fucking money
I'm all in favor of this but part 1 has been made politically toxic for decades, and part 2 only seems to be taken seriously when the Other Guys are in power.
For example, in the US I simply cant take the Republican party's fiscal opinions seriously when they bleat about the debt on the one hand and yet dont blink when asked to fund another foreign war.
republicans are much much bigger spenders than democrats by miles. they are “anti socialism” publicly but are 2nd largest socialist party on the planet (behind china).
"Socialism is when the government gives tax breaks to the rich and destroys public institutions". And calling China "socialist" is rich when it's probably the most ruthlessly capitalistic nation on the planet, where worker unions are straight up illegal.
The West saw its golden age when redistributive taxation was maximum. We have to get back to that, or the country will continue to agonize under crumbling infrastructure and failing institutions.
What intervention could <roll back the industrial revolution>?
On the technical side, perhaps none. But countless people in the labor movement spilled blood for their rights and dignity. Maybe something similar will be needed here, too.
So because a billionaire donated to this pac that means “billionaires” as a group are responsible for this? That’s like saying black people are responsible for electing Trump because >= 1 black person voted for him.
Your ignorance continues. Some of the billionaires fund it via intermediary organizations like Heritage Foundation. They are quite organized and not only acting as individuals.
Thanks for the links, I'll checkout the Generic Cell Rate Algorithm!
reply