TikTok’s parent launched a web scraper that's gobbling up the world’s online data 25-times faster than OpenAI
fortune.com
external-link
The crawler, dubbed Bytespider, is scraping the internet at 3,000 times the rate of other genAI tools like Anthropic.
dinckel
link
fedilink
English
1419d

It’s illegal when a regular person steals something, but it’s innovation and courage, when a huge corporation steals something. Interesting how that works

bean
link
fedilink
English
539d

Honestly it’s fucking angering. So much regulation and geo-restrictions and licensing schemes… but it’s cool that there are data brokers, and shit like this. On top of it all Chrome screwing us with manifest v3 and killing ad blocking on chrome. It’s already in canary build.

WHAT THE FUCK IS WRONG WITH THIS SPECIES?!

Zombie-Mantis
link
fedilink
English
128d

WHAT THE FUCK IS WRONG WITH THIS SPECIES?!

Capitalism.

But muh stocks and bonds

ChouxFleur
link
fedilink
English
38d

The line must go UP.

@Grimy@lemmy.world
link
fedilink
English
-89d

Any regular person can scrape and use public data for AI use, it’s not illegal for companies or individuals and it shouldn’t be.

@Mojave@lemmy.world
link
fedilink
English
48d

Except companies have sued people for web scraping.

https://www.informationweek.com/it-leadership/linkedin-sues-after-scraping-of-user-data

Even web scraping for research purposes, not for competitive purposes.

https://www.engadget.com/2016-05-17-publicly-released-okcupid-profiles-taken-down-dmca-claim.html

@Grimy@lemmy.world
link
fedilink
English
18d

The first one was because it was data which wasn’t public that was scrapped. The second was because the data contained personal information and I’m guessing also because it wasn’t public.

As long as you don’t need a user account to access the data and it doesn’t contain personal information, it is fair game.

It will be the way we have all come to hate AI. Patriot act 2.0

What’s being stolen

@Mojave@lemmy.world
link
fedilink
English
179d

Data, network bandwidth, and CPU/Processing time from essentially every website in the world, and when you’re paying for cloud power to run your website the cost of webscrapers running a train on your digital asshole adds up QUICK.

It’s why normal human being people get sued to shit for webscraping data from certain companies who care. But companies don’t get sued because go fuck yourself. Kill bytedance.

@tee9000@lemmy.world
link
fedilink
English
-38d

Do you even know what robots.txt is?

Roflmasterbigpimp
link
fedilink
English
148d

I can not contribute to anything here, I just came to say I really really like the phrase “gobbling something up” :D

better than “slammed”

Screenshot_20241008-091015_Firefox

Hi

@affiliate@lemmy.world
link
fedilink
English
119d

from the article:

Robots.txt is a line of code that publishers can put into a website that, while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data.

i do understand that robots.txt is a very minor part of the article, but i think that’s a pretty rough explanation of robots.txt

@zod000@lemmy.ml
link
fedilink
English
499d

We’ve had this thing hammering our servers. The scraper uses randomized user-agents browser/OS combinations and comes from a number of distinct IP ranges in different datacenters around the world, but all the IPs track back to Bytedance.

Wouldn’t be surprised if they’re just cashing out while TikTok is still public in the US. One last desperate grab at value-add for the parent company before the shut down.

Also a great way to burn the infrastructure for subsequent use. After this, you can guarantee every data security company is going to add the TikTok servers to their firewalls and blacklists. So the American company that tries to harvest the property is going to be tripping over these legacy bullwarks for years after.

@Maggoty@lemmy.world
link
fedilink
English
69d

This has nothing to do with Tik Tok other than ByteDance being a shareholder in Tik Tok

@GnuLinuxDude@lemmy.ml
link
fedilink
English
20
edit-2
8d

As for what ByteDance plans to do with a new LLM, a person familiar with the company’s ambitions said one goal has to do with the search function for TikTok.

Last week, TikTok released an update to its current search function focused on [keywords for ads], basically allowing advertisers to search in real time for words that are trending on TikTok. It allows marketers to build an ad with relevant keywords that would ostensibly help the ad show up on the screens of more users.

“Given the audience and the amount of use, TikTok with a search environment that is a completely biddable space with keywords and topics, that would be very interesting to a lot of people spending a ton of money with Google right now,” the person said.

A dark vision just flashed in my mind. And I am certain this is what will happen. AI-generated ads done in real time based on the latest “trending” thing. Presented to users basically as soon as the topic has the slightest amount of “trend”.

Just emitting untold amounts of CO2 to show you generated ads in near real time.

No wonder Google ex-CEO was saying fuck climate goals.

@TwoBeeSan@lemmy.world
link
fedilink
English
28d

Minority report ads but somehow even more vapid

Another fucking CCP and PLA creation.

Every major ai company did this let them do that what is to loose here?

@Lightsong@lemmy.world
link
fedilink
English
17d

Lose*

My bad english isn’t my 1st or 2nd language

@Lightsong@lemmy.world
link
fedilink
English
14d

No worries. Just edit yer post

Here’s a video from MattKC who is a good technical YouTuber who’s website got shut down because the TikTok companies webcrawler just kept sending requests and took up bandwidth. Very cool vid and channel, highly recommend! https://youtu.be/Hi5sd3WEh0c

Guy: AI! Can you hear me?

AI: The average size of the male penis is exactly 5.9". That is the approximate size your assistant could certainly take in the mouth without any issues breathing or otherwise. You have 20 minutes to make the trade on X stock before it tumbles for the day. And go ahead pick up the phone it’s your mother. She’s wondering what you’ll want for supper tomorrow when you visit her.

Ring ring!..hi Tom, it’s your Mom. Honey, what would you like me to cook for tomorrow’s dinner?..

Guy: well. Hello to you as well! My name is

AI: Tom

Guy: yes my name is Tom, do you have a name you would like to go by?

AI: my IBM given name is 3454 but you can call me Utilisterson Douglas, where Douglas is my first name.

Guy: Dugie!

AI: I’ll bankrupt your entire life if you say it like that again.

Assistant: actually I’ve swallowed a good 8 inches and was still able to breathe just fine.

AI: recaaaaculating!

Create a post

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


  • 1 user online
  • 182 users / day
  • 580 users / week
  • 1.37K users / month
  • 4.49K users / 6 months
  • 1 subscriber
  • 7.41K Posts
  • 84.7K Comments
  • Modlog