☆ Yσɠƚԋσʂ ☆
  • 1.29K Posts
  • 1.07K Comments
Joined 6Y ago
cake
Cake day: Jan 18, 2020

help-circle
rss
The Knowledge Graph of Thoughts is a new architecture for AI assistants that makes them both cheaper to run and better at tough problems. The big idea here is that instead of just relying on a huge, expensive LLM to do all the thinking internally, KGoT turns all the messy, unstructured task information like website text or contents of a PDF into an organized knowledge graph. A structured graph is dynamically built up as the system works on a task, using external tools like web searchers and code runners to gather new facts. Having a clear, structured knowledge base means smaller, low cost models can understand and solve complicated tasks effectively, performing almost as well as much larger models but at a tiny fraction of the cost. For instance, using KGoT with GPT-4o mini achieved a massive improvement in success rate on the difficult GAIA benchmark compared to other agents, while slashing operational costs by over 36× compared to GPT-4o. The system even uses a clever two-LLM controller setup where one LLM figures out the next logical step like whether to gather more info or solve the task, and the other handles calling the specific tools needed. Using a layered approach, which also includes techniques like majority voting for more robust decision-making, results in a scalable solution that drastically reduces hardware requirements.
fedilink


It’s powered by a miniature nuclear reactor meaning that it can stay airborne long enough to fly around the world and approach targets from any direction.


You gotta love it when people start commenting on a topic they have no clue about. There is no reentry, this is a low flying missile. The whole point of it is that it’s a loitering missile that can fly around for months on end. That’s the whole reason for the panic in NATO, it’s not possible to track it at all. Time for you to stop embarrassing yourself in public.


What actually turned out to have the capabilities of wet farts were all the fabled NATO weapons that were sent to Ukraine. Every game wunderwaffe that was supposed to turn the war right around turned out to be a dud in the end.



AI is being developed in China in a very different way from the west because the material conditions in China are different. https://dialecticaldispatches.substack.com/p/the-ghost-in-the-machine

The reason Chinese companies release LLMs as open source isn’t actually confusing either. It’s not being treated as a product, but rather as foundational technology that things will be built upon. Think of it the same way as the current open source infrastructure that underpins the internet. Most companies aren’t trying to monetize Linux directly, rather they use it to build actual products on top of.

However, dragging the US into a tech race it can’t win is also a factor whether it’s done intentionally by China or not. https://dialecticaldispatches.substack.com/p/the-ai-race-isnt-about-chips-its



https://archive.ph/20251109191103/https://www.bloomberg.com/opinion/articles/2025-11-09/how-much-of-silicon-valley-is-built-on-chinese-ai
fedilink










Nested Learning: A new ML paradigm for continual learning
A new paper argues that current LLMs are fundamentally broken because they're completely static. They call it "anterograde amnesia", which is honestly spot on. A model gets pre-trained, and from that moment on, its weights are frozen. It can't actually learn anything new. Sure, it has a context window, but that's just short-term memory. The model can't take new information from its context and permanently update its own parameters. The knowledge in its MLP layers is stuck in the past, and the attention mechanism is the only part that's live, but it forgets everything instantly. The paper introduces what they term Nested Learning to fix this. The whole idea is to stop thinking of a model as one big, deep stack of layers that all update at the same time. Instead, they take inspiration from the brain, which has all kinds of different update cycles running at different speeds in form of brain waves. They represent the model as a set of nested optimization problems , where each level has its own update frequency. Instead of just deep layers, you have levels defined by how often they learn. The idea of levels was then used to extend the standard Transformer which has a fast attention level that updates every token and the slow MLP layers that update only during pre-training. There's no in-between. The paper presents a Hierarchical Optimizers and Parallel Extensible model with additional levels. You might have a mid-frequency level that updates its own weights every, say, 1,000 tokens it processes, and a slower-frequency level that updates every 100,000 tokens, and so on. The result is a model that can actually consolidate new information it sees after pre-training. It can learn new facts from a long document and bake them into that mid-level memory, all while the deep, core knowledge in the slowest level stays stable. It creates a proper gradient of memory from short-term to long-term, allowing the model to finally learn on the fly without just forgetting everything or suffering catastrophic forgetting.
fedilink



Frankly, I’ve never really understood the logic of bailouts. If a company is not solvent, but it’s deemed to be strategically important then the government should simply be taking a stake in it. That’s what would happen on the private markets with another company buying it out. The whole notion that the government should just throw money at the failing companies with no strings attached is beyond absurd.










https://web.archive.org/web/20251106084318/https://www.scmp.com/news/china/science/article/3331031/china-unveils-power-thorium-reactor-worlds-largest-cargo-ship
fedilink

I mean if you have a verifiable set of steps that build from the answer to first principles, that does seem to enable trust worthiness. Specifically because it makes it possible for a human to follow the chain and verify it as well. This is basically what underpins the scientific method and how we compensate for the biases and hallucinations that humans have. You have a reproducible set of steps that can explained and followed. And what they’re building is very useful because it lets you apply this method to many problems where it would’ve been simply too much effort to do manually.


It’s like watching a grand master play chess with a toddler.









This paper comes up with a really clever architectural solution to LLM hallucinations, especially for complex, technical topics. The core idea is that all our knowledge, from textbooks to wikis, is "radically compressed". It gives you the conclusions but hides all the step-by-step reasoning that justifies them. They call it a vast, unrecorded network of derivations the "intellectual dark matter" of knowledge. LLMs being trained on this compressed, conclusion-oriented data is one reason why they fail so often. When you ask them to explain something deeply, they just confidently hallucinate plausible-sounding "dark matter". The solution the paper demonstrates is to use a massive pipeline to "decompress" all of the steps and make the answer verifiable. It starts with a "Socrates agent" that uses a curriculum of about 200 university courses to automatically generate around 3 million first-principles questions. Then comes the clever part, which is basically a CI/CD pipeline for knowledge. To stop hallucinations, they run every single question through multiple different LLMs. If these models don't independently arrive at the exact same verifiable endpoint, like a final number or formula, the entire question-and-answer pair is thrown in the trash. This rigorous cross-model consensus filters out the junk and leaves them with a clean and verified dataset of Long Chains-of-Thought (LCoTs). The first benefit of having such a clean knowledge base is a "Brainstorm Search Engine" that performs "inverse knowledge search". Instead of just searching for a definition, you input a concept and the engine retrieves all the diverse, verified derivational chains that lead to that concept. This allows you to explore a concept's origins and see all the non-trivial, cross-disciplinary connections that are normally hidden. The second and biggest benefit is the "Plato" synthesizer, which is how they solve hallucinations. Instead of just generating an article from scratch, it first queries the Brainstorm engine to retrieve all the relevant, pre-verified LCoT "reasoning scaffolds". Its only job is then to narrate and synthesize those verified chains into a coherent article. The results are pretty impressive. The articles generated this way have significantly higher knowledge-point density and, most importantly, substantially lower factual error rates, reducing hallucinations by about 50% compared to a baseline LLM. They used this framework to automatically generate "SciencePedia," an encyclopedia with an initial 200,000 entries, solving the "cold start" problem that plagues human-curated wikis. The whole "verify-then-synthesize" architecture feels like it could pave the way for AI systems that are able to produce verifiable results and are therefore trustworthy.
fedilink

Think of it this way, the investors are basically like people going to a casino. They start with a bunch money, and they start losing that money over time. That’s what’s happening here. Right now, they still haven’t lost enough money to quit playing, they still think they’ll make their investment back. At some point they either run out of money entirely, or they sober up and decide to cut their losses. That’s what’s going to change between now and when the bubble starts to pop. We simply haven’t hit the inflection point when the investors start to panic.


It does actually

The economic nightmare scenario is that the unprecedented spending on AI doesn’t yield a profit anytime soon, if ever, and data centers sit at the center of those fears. Such a collapse has come for infrastructure booms past: Rapid construction of canals, railroads, and the fiber-optic cables laid during the dot-com bubble all created frenzies of hype, investment, and financial speculation that crashed markets. Of course, all of these build-outs did transform the world; generative AI, bubble or not, may do the same.

The scale of the spending is absolutely mind blowing. We’re talking about $400 billion in AI infrastructure spending this year alone, which is like funding a new Apollo program every 10 months. But the revenue is basically pocket change compared to the spending.

As the article notes, the reality check is already happening.

Much is in flux. Chatbots and AI chips are getting more efficient almost by the day, while the business case for deploying generative-AI tools remains shaky. A recent report from McKinsey found that nearly 80 percent of companies using AI discovered that the technology had no significant impact on their bottom line. Meanwhile, nobody can say, beyond a few years, just how many more data centers Silicon Valley will need. There are researchers who believe there may already be enough electricity and computing power to meet generative AI’s requirements for years to come.

The whole house of cards is propped up by this idea that AI will at some point pay for itself, but the math just doesn’t add up. These companies need to generate something like $2 trillion in AI revenue by 2030 to even break even on all this capex, and right now, they’re nowhere close. OpenAI alone is burning through cash like it’s going out of style, raising billions every few months while losing money hand over fist.

I expect that once it’s finally acknowledged that the US is in a recession, that’s finally going to sober people up and make investors more cautious. The VCs who were happily writing checks based on vibes and potential will start demanding to see actual earnings, and that easy money environment that’s been fuelling this whole boom is going to vanish overnight.

When a few big institutional investors get spooked and start quietly exiting their positions, it could trigger a full blown market panic. At that point, we’ll see a classic death spiral. The companies that have been living on investor faith, with no real path to profitability, are going to run out of cash and hit the wall leading to an extinction level event in the AI ecosystem.

If tech stocks fall because of AI companies failing to deliver on their promises, the highly leveraged hedge funds that are invested in these companies could be forced into fire sales. This could create a vicious cycle, causing the financial damage to spread to pension funds, mutual funds, insurance companies, and everyday investors. As capital flees the market, non-tech stocks will also plummet: bad news for anyone who thought to play it safe and invest in, for instance, real estate. If the damage were to knock down private-equity firms (which are invested in these data centers) themselves—which manage trillions and trillions of dollars in assets and constitute what is basically a global shadow-banking system—that could produce another major crash.

When that all actually starts happening ultimately depends on how long big investors are willing to keep pouring billions into these companies without seeing any return. I can see at least another year before reality starts setting in, and people realize that they’re never getting their money back.






Again, this is a very US centred perspective. I highly urge you to watch this interview with the Alibaba cloud founder on how this tech is being approached in China https://www.youtube.com/watch?v=X0PaVrpFD14


You’re such an angry little ignoramus. The GPT-NeoX repo on GitHub is the actual codebase they used to train these models. They also open-sourced the training data, checkpoints, and all the tools.

However, even if you were right that the weights were worthless, which they’re obviously not, and there were no open projects which there are, the solution would be to develop models from scratch in the open instead of screeching at people and pretending this tech is just going to go away because it offends you personally.

And nobody says LLMs are anything other than Markov chains at a fundamental level. However, just like Markov chains themselves, they have plenty of real world uses. Some very obvious ones include doing translations, generating subtitles, doing text to speech, and describing images for visually impaired. There are plenty of other uses for these tools.

I love how you presumed to know better than the entire world what technology to focus on. The megalomania is absolutely hilarious. Like all these researchers can’t understand that this tech is a dead end, it takes the brilliant mind of some lemmy troll to figure it out. I’m sure your mommy tells you you’re very special every day.


You seem to have a very US centric perspective on this tech the situation in China looks to be quite different. Meanwhile, whether you personally think the benefits are outweighed by whatever dangers you envision, the reality is that you can’t put toothpaste back in the tube at this point. LLMs will continue to be developed. The only question is how that’s going to be done and who will control this tech. I’d much rather see it developed in the open.


There is a lot of hype around LLMs, and other forms of AI certainly should be getting more attention, but arguing that this tech no value is simply disingenuous. People really need to stop perseverating over the fact that this tech exists because it’s not going anywhere.


It’s worth noting that humans aren’t immune to the problem either. The real solution will be to have a system that can do reasoning and have a heuristic for figuring out what’s likely a hallucination or not. The reason we’re able to do that is because we interact with the outside world, and we get feedback when our internal model diverges from it that allows us to bring it in sync.




In a major leap for the global semiconductor industry, a joint Chinese research team has developed a method that can slash defects in lithography – a critical step in chipmaking – by up to 99 per cent. The researchers achieved unprecedented clarity by using cryo-electron tomography (cryo-ET) to pinpoint, for the first time, the minute sources of common manufacturing flaws. The findings, published in the journal Nature Communications on September 30, by Professor Peng Hailin from Peking University in collaboration with researchers from Tsinghua University and the University of Hong Kong, were hailed by reviewers as a “fancy tool” that “would benefit peer researchers and industrial users quite a lot”. “The team has proposed a solution compatible with existing semiconductor production lines,” Peng said in an interview with Beijing-based Science and Technology Daily published on Monday. “It can reduce lithography defects on 12-inch (30cm) wafers by 99 per cent,” he added, indicating substantial cost benefits to the market. Lithography is one of the most critical steps in chip manufacturing. “It can be understood as ‘printing circuits’ onto semiconductor wafers such as silicon,” Peng said. “Essentially, an ultra-precise ‘projector’ shrinks and transfers pre-designed circuit patterns onto a special film coating the wafer, which is then developed and fixed.”
fedilink







Yeah, they mention it in the vid. It’s the exact same principle, but this way you can do it anywhere without needing a large body of water near by.


The approach China is taking is to invest in all kinds of different approaches, and then see what works. I imagine the answer is going to be that different types of energy storage will work best in different situations. Something like gravity storage might be useful for balancing short term fluctuations in the grid, it can be built anywhere, and it’s very safe.


These things only make sense at very large scale. China has already built some and they’ve approved more projects going forward. https://www.enlit.world/library/china-connects-gravity-storage-and-launches-three-new-projects






Aww muffing, you keep on coping there. Meanwhile, I didn’t compare anything, I gave an example of how building a particular thing over and over brings the costs and time of construction down. The fact that you still can’t comprehend that how little goes on in that head of yours.


Concrete is certainly a lot more clean than lithium mining. Meanwhile, construction of specific things is obviously something that you learn to do better over time. For example, here’s how cost of nuclear reactor construction has dropped in China as they learned from building them. Absolutely incredible that the concept of getting better at doing things through practice escapes you.

Meanwhile they are already in production, hilarious how you didn’t even bother checking a link past 2018 before spewing more drivel. 🤣

Following the start of grid interconnection in September 2023, the 25MW/100MWh EVx GESS in Rudong achieved full interconnection after the completion of the final 4km 35kV overhead power line to a remote end substation, as planned with local state grid authorities.


Obviously much cheaper than mining lithium, and the expense goes down as you scale out the technology as China is doing. But you keep on coping little buddy.





It could be that it did happen before, but it was just individual cases and if the outage wasn’t long then probably wasn’t noteworthy. This time it was a whole bunch of people affected all at once for a prolonged period. And you’re likely right that there’s probably a series of states the device can be in, and it does calls to AWS as it moves through them, so probably got stuck at a particular stage and couldn’t move forward cause it couldn’t talk to the mothership.


Yeah, it’s almost certain everybody would just end up using the same library.



It’s kind of amazing to think about how none of these products are designed to have a fallback functionality during an outage.


Yeah, I’m mostly excited about LLMs that can be run locally. And it really does look like there’s a lot more optimizing that can happen going forward. Stuff like this is also really exciting. It’d be pretty amazing if we get to the point where models that perform as well as current 600+ billion parameter ones could run on a phone.


turns out a picture is really worth a thousand words :)