Experimenting with ACE-step

Yeah, I've gone to the dark side.

I said I was messing around with AI music creation via ACE-step. That one is a bit more towards a diffusion model, so (at the moment) less controllable than Suno, but potentially more flexible. They boast about the number of languages it can handle...but at the same time, admit that they really haven't trained it on opera, or (as my own small experiments seemed to show) much of anything at all outside of pop music.

(They also side-step around the sound quality, with the user base developing magic incantations that might be improving things if you squint, or might just be random "sometimes it doesn't sound quite so bad" and has nothing to do with their "and set cfig to 1.14" stuff.)

The one that I actually wanted to talk about was the attempt to do a villain song/"I want" song. The result is less Broadway, more Billie Eilish. Among other things, ACE doesn't know how to belt. Or like female singers in a range other than light soprano. It's understanding of "chorus" v. "verse" is "sing louder!" It also only takes lyrics, like anything else, as suggestions. 

But on the fourth or fifth run it came out with this weird melisma that's almost Sondheim or something. Pity it couldn't carry through. I could improvise a better chorus (it is an absolute natural to sequence this one, half-stepping higher and higher). Plus, at least one previous run was smart enough to do the intro on solo piano. I like the "is that supposed to be an oboe?" but it should enter later.

Like I said. Not really trained on showtunes. That's why it surprised me that this run, instead of repeating the chorus over and over, drifting further and further from the melody as it goes (and not even a Truck Driver Gear Shift!) it went for an extended instrumental break. Cool. On stage, this would be a dance, or perhaps an aerial dance.

The less said about the Wan2.1 animations, the better. At some point in the increasingly large and power-hungry tools you can script better, and start using LORA to keep the animator from going completely off model. At this level it is just as well I've got a 3060 ti with 12Gb that is practically a potato when it comes to video -- because Wan2.1 also loses the plot after 60-90 frames of video. So no Bryce-length render times. It also has some memory leaks so even if I could automate a batch, the second one would fail.

But on the other hand; this gave a nice chance to demonstrate using img2img mode for composing a Stable Diffusion image. I did those roughs in Paint3D, using my knowledge of how SD "sees" an image. The airship took a bit of inpainting as the AI really wanted to make either a blimp or a clipper, not try to get both in the same machine. The others I used simple convergence; rendered a half-dozen versions, picked one, then re-rendered on that one to zero it in.

At least with SDXL you can prompt a little. ACE-step and WAN2.1 go even more nuts about stray words -- my first attempt at the airship had "flying" in the prompt and the AI added wings to it. For the ACE renders, my prompts were as short as four words long.

The other lesson here is that the AI will willingly change lots of things, but it is real stickler about proportions. What it saw in the comp, it will keep. A big-headed guy with a short-necked guitar is going to stay that way through the generations, and it is really difficult to fix it once you are in the clean-up render phase.

Fox and goose

Spent a chunk of the weekend wrestling with mostly that class of problem. Now I'm outlined to the end. I know; I was outlined before. Now I have more detail.

For me, it isn't figuring out why there's a bomb on the train. Or how the heroes figure out that there is a bomb on the train. It is all about controlling when they find out. That's the part that annoys me and feels like it takes a lot more time than it should.

That's because the why is rooted in the scene in which that clue is dropped, and the ordering of the scenes is at the mercy of geography, time, and all the other plot threads that led to them finding the briefcase full of blasting caps. Like, the briefcase was dropped by the Crooked Man during the chase scene, but that means he was heading away from the station, so how did he end up back on the train?

Multi-value problems. 

Weirdly, this wraps all the way back to one of the basics. Having a bat-computer moment, "I just realized there's a bomb on the train!" is telling. You want the moment that information arrives to be dramatic, and that is practically synonymous with showing it.

Which is to say, it is the central business of a scene. Which makes the scenes inseparable from the clues, which means you have to figure out how to do them in the one order that doesn't end up with a fat chicken and an even fatter fox.

My concentration wasn't up for that for most of the last few days so I made progress on the new cover/series rebrand. I'd roughed out a logo but I didn't quite have the patience for dragging vectors around. And the font I used ("Adventure") isn't licensed for commercial use.

Took a chance on a guy in Fiverr who did not quite share enough English with me to make communication fluid, but seemed familiar with the style I wanted. Nobody in this business makes sketches anymore. The worst are the cover designers. They give you a finished cover and besides the limited number of revisions they allow, you feel bad about asking them to cut into a finished artwork.

That is because of the pipeline, which is heavily asset-based, and I understand. There's nothing wrong about having a cover that looks like other covers in the same genre. It makes it easier for the reader to find the book, and it give the reader a certain trust that they are buying the kind of book they wanted to be buying.

It can lead to asset over-use, though.

My Fiverrr guy was showing me full-color graphics. So, fine, maybe he liked sketching with paint tools. The order was explicitly for vector graphics, though. Was he going to turn this more cartoony paint into proper vectors at the last step?

Turned out, that was the finished art. The vectors were him throwing blobby paint sketch into auto-convert, making one of those impossible-to-edit vector "bit clouds."

So I'm doing the vectors after all. Sigh. Well, half the reason I hired an outsider is because I'm at best a craftsman and I wanted the imagination of an artist. And I liked his ideas. But that's left me even less eager to hire other people.

Because you are up against efficiency. They don't want to do your ideas; they want to do the idea that can be done with stock assets, quick-and-dirty "convert to vector," and more and more, AI. And on the way to talking you into picking that instead (via making a really glossy final version they hope you will be attracted to enough to go with despite reservations), talk you up into Facebook banner and animated logo and an AI image of a hand holding your book.

I already dropped a hundred on a cover I don't want to use. I may just have to go back to doing my own.

In a cavern in a canyon...

...excavating for bitcoins?

I'm not really where I want to be as a writer to tackle the New Orleans story. But more to the point, I've got this weird attraction towards the idea of sending Bill Bixby///Penny to Colorado or Montana or something, find a low-population bit of near-wilderness and put a massive super-advanced tech center for her to get in trouble in.

So, like, that new Google AI center, only located in an old coal mine. Or something else suitably techno-thriller epically over-scaled and real-world commercially implausible.

I can find a remote mountain or a salt mine or something else sufficiently crazy. I can invent a biotech firm or a nanotech lab or some other appropriately dangerous and dystopian technology.

What I'm having trouble with, is tying it to the conceits of the Athena Fox series. Which are that there is a subject of history somewhere in the mix (preferable a narrow or easy-to-define band in time and/or space). That there is an archaeological issue to raise, like archaeological ethics or the problem with field schools. And that at some point it goes Tomb Raider, in that there is some perfectly sensible reason why Penny finds herself crawling down booby-trapped tunnels or fighting with ninja.

For the Bill Bixby stories, she doesn't need a good excuse to be there. There's no dig involved. She got off the Greyhound somewhere in middle America. Or got a phone call.

Yeah, like I'm in a good place to be plotting a new book. I'm having shiny new "I'd rather be writing anything but this" syndrome. Actually, I feel okay about The Early Fox. It went different than I expected, but it looks like I'm hitting the beats I hoped to hit.

I'm enjoying writing it, it is just a lot of work and I'm having trouble coming up with the sheer concentrated brainpower to deal with the stuff that's outlined.

The Atlas-F silo won't be bad, I can stumble through the next Freeman conversations, the long chase down the Jornada de Muerto has complexities but it isn't anything I haven't done before.

No, but going to the rez is challenging. Not looking forward to tackling that. Worse, I decided I need to send her to the War Zone (to contact an urban explorer who can tell her about Lon's errand at the Atlas-F silo outside of Roswell). Which almost certainly includes cops, and that means the deputy sheriff at Alamogordo is firmly pencilled in now. 

And I still have to go back and do the nuke museum scene. The museum is fine. The "Los Alamos Wife" bit is going to take a little more. Probably not watching a full season of Manhattan. Fortunately, the museum has free online videos that are somewhat more...historical.

I, for one, welcome our new...

...clipping service?

Yes, AI is a fancy plagiarism machine. But that doesn't make it entirely useless as a sort of clipping service; it goes out there and finds things that are like what you seem to be describing, shamelessly rips them off and shows you the resulting pastiche.

Which you then get independently inspired by and do your own, only vaguely similar, thing in response.

Since I went ahead and put ComfyUI -- it of the many nodes -- on my gaming machine (it's the one with the fast graphics card), I tried out text-to-music.

I am not impressed. Besides all the usual artist-type comments about how it doesn't understand anything about rhythm or harmony, and the front end was written by someone who doesn't know a fugue from a flugelhorn (and may have never even heard of the latter), the technical side of me was appalled.

The sound quality sucks rocks. It is damned near unlistenable.

Still, a couple of runs with the prompt "steampunk sea chanty" were amusing to listen to and tempted me for a long moment to go break out the more traditional tools.

This particular implementation (not that ACE is getting top marks in anyone's text-to-music books) totally sucks rocks. And not in a good way.

And this is technically interesting from two totally different directions. One is that getting a glossy appearance is one of the things AI is really good at (too good; it gets called out for that too-perfect "AI look" all the time). So why is ACE so bad? And is this endemic across using diffusion models for music creation?

For that matter, are there competing technologies and if so, what are they? I mean, we've been doing computationally generated music since before there were computers. With the robust ecosystem of software synthesis and sound manipulation, attacking the problem Band in a Box style (which was a pattern-fill sort of MIDI generator) seems more fruitful.

The other is that this seems so very...useless. Drawing is hard. It takes a lot of time to learn and even if you are good and using tools that are fast (some of which highly leverage computing in various ways) it still takes time to do.

Punching a keyboard controller is so much less effort in comparison. The barrier to music creation at home, now that there are things like Garage Band being given away with the operating system, seems very short to me.

But then, isn't that what AI is basically making a name for itself with? Taking tasks you didn't really need to automate, and automating them anyhow?

***

In more productive news, I made a full mock-up of my "map with red line" cover concept. I like it.

Since I'm not that interested in wrestling vectors for days, and I started with a non-commercial font in the first place, I reached out to Fiverrrrr to see if anyone would do the running series title graphic for me. Next step is taking it to my cover artists and asking for the "clean up my cover" service. They might be willing to do the font work, but they also might cheapen out, and it is a major design element for me. The rest of it is a lot of tweaking font fills and kerning (whatever the hell kerning is. I joke. I do know what kerning is). And that sort of thing, they can do just fine.

I'll see. Right now Fiverr is being Fiverr. People missing the point as they frantically run their AI translation software in the background, hoping that being able to pretend to share a language is all they need to pretend to share background and aesthetics as well...and relentlessly hoping to upsell me into an entire cover-and-book-edit while they are at it.

***

ComfyUI's implementation of the WAN2.1 also ran, but illustrated a very basic problem with the current paradigm of AI. No, the belief of legions of middle-management to the contrary, plain-language is not the appropriate tool to communicate technical needs.

We do have a vocabulary of shots and cinematography but that isn't how shots, at the level of a film, are constructed and communicated. There's story-boards, for one. And, yes, there is something like that moving in.

Look, better analogy. I've worked around a bunch of choreographers. They will often demonstrate moves, and work through them and count them out while teaching them. They don't just say, "It's just a jump to the left, then a step to the right."

So shoving 250 words into a text prompt is not going to create the combination of actor move and camera move that the eye of an artist desires. It can only make a shot similar to the shots it has seen other people do, that might look good enough to convince someone without those skills that it is communicating the directorial intent.

Is nano banana qwen?

Actual title of a post on the StableDiffusion subreddit.

My Automatic1111 install croaked again. "Of course!" say the power users on the SD forums. "That's ancient. They stopped supporting it in mid-2024!"

That's how fast AI is moving. Yes, people are getting water-cooled Nvidia 5060 rigs set up (under an 16-core Ryzen or better). ComfyAI is old hat, with several other forks of Automatic1111, other independent run environments, and a front end for Comfy (which itself is basically a front end) that's got six different versions out there already.

Yeah, so I installed Comfy, even if it, too, is already old hat. But since another thing that is old hat is doing a clean install of Automatic1111 I just did that, too. Maybe this time I'll be smart and clone a backup (without the models, which is what takes up all the space).

It is old but I more or less know how to use it. It also is apparently much superior to ComfyUI for inpainting, and the thing I'm having fun with (I can't really call it workflow, since -- cue the Jurassic Park quote -- I'm not actually working on anything) is a process that is basically hyper-inpainting.

On a line with "assisted by software" towards one end and "created by the machine" on the other, I take some minor comfort that I'm leaning closer to the former than you might expect from the phrase "using AI." But after all, that's what editing is now over in book land; you are using AI tools, but they identify possible grammatical errors, and if you let them, make suggestions. You chose what to do in response. And "letting the AI change it" isn't the most common response.

This doesn't take away from the basic problems of AI. The morality (and possible legality) of the training data. The dangers of it, and not just misuse. And lurking under all of that, the black box nature of the thing. You are never quite in control and you never quite know what you are going to get, and for all that I could go on about serendipity (Rodin knocking a bust off the table and turning it into Man with a Broken Nose) and the ornery nature of organic tools (you never have total control over where that ink will go or how that paint will look)...

***

Revisited the cover concept again with a hybrid of Indiana Jones map and a bit of the Rivers of London look. That's basically a map, with the Beau Geste dotted line, the series running header as the dominant text element in swooping "adventure" font, and a compass rose with book number in the center.

Meanwhile writing is going slowly. I did a thousand words on the next scene yesterday. But even though it works, I'd prefer something with more conflict in it. So complete rewrite of that one.

I have a lot of Maytag Repairman time at work (we did an install yesterday; out of the building and something constructive to do, yay!) I can't concentrate well enough to write, so been reading up at (and on) Atomic Rockets.

Turns out Project Pluto wasn't even the height of insanity. Try Nuclear Salt Water rocket engine. That thing scared Scott Manley! And then there's what I can only describe as a nuclear solid rocket booster, for which one of the more interesting limitations is they have to make them really long and thin because there is no way of slowing down the China Syndrome melt-down which is heating up the lithium hydroxide reaction mass! 

I've also decided to not worry about lost opportunities for extra conflict and drama. The Athena Fox stories are more of a free-form exploration of a place, culture and time, where the solution of the mystery arises organically from the environment.

A plot which is intensively "Fetch Quest" oriented? Maybe The Tiki Stars is a good place to try that out.

Buy a bago, win-a-bago

You win a few, you lose a few.

I was looking up and down 54 and thinking about ghost towns. There's a cool one close to Orogrande (which itself is pretty weird). I got tempted by thoughts of abandoned silver mining towns, by Old West associations, or even by graveyards (cue "Ecstasy of Gold.")

But in the end, I didn't need to do a whole scene about Penny turning around the truck (and sending a text to Lon). I can just pick up with her meeting.

So I move on to thinking about my Alamogordo sources, but it felt a little too grand tour to send her to every notable thing in that town. And then I realized I could just set this upcoming scene at the burger joint. That hits a couple of useful plot bits; this changes Penny's relationship with "Dynel," and means the sheriff can say "You were seen in his company" when he goes asking around. And I do love a hub location. I sent most of the cast to the cafe in the last book, enough to make it a running gag.

Besides, I'm saving the giant pistachio.

It ain't all winning, though. At the same time, I realized I really need to go Oop North to interview this one guy. It wasn't right to leave Native Americans out of the story of White Sands, or the atomic age in New Mexico, so I had that in the story. And now that I've involved NAGPRA and have several key native characters, I have to do it. I have to send the story to the rez.

That does give me more freedom to focus in. To basically drop any cowboy stuff (another book, right). 

Largely, this is still moving to third-act territory, where all the basic settings and the cast are all here, and the reader won't have to go on learning new things. It makes sense to focus in.

I was also increasingly feeling I needed to focus in on New Mexico, and Egtved Girl didn't make the cut. I decided I was happy losing all of those vignettes except for bookend scenes; "White Sands Girl" (I still can't figure out which exact numbered trackway she was, but she was the one moving quick across mud, carrying a child, and encountering both a columbian mammoth and a Hanson's ground sloth. Busy day.)

And Valentina. Who encountered a whole hell of a lot of nothing (more nothing than most of us will ever experience) but had an even busier day.

But at the same time, the museum scene had been evolving (I haven't written it yet, even though it happens quite a bit further back than where I currently am). And I thought of a cool thing I could do that would both pull the threads together, and fill out the necessary space for Penny's epic hike through White Sands from I-haven't-decided-yet to the Trinity Site.

So...Lozen out. Lozen in. And Egtved Girl might still make the cut. But...the way this is being done, it takes a lot of the pressure I was feeling about having to do enough research to pull off 1-2,000 words on "woman walking" at various random points of history.

In any case, I'm pretty much plotted through to the end now. I've stopped updating the outline, but I have place-holder chapters out through to the end of the current act. Now I just need to find the focus to write (the hot weather is not helping!)