Haskell and functional programming

This is quite a good intro:
http://www.cs.ou.edu/~rlpage/fpclassCurrent/textbook/haskell.shtml

It is more of a lesson in programming in a functional style using haskell than a lesson in programming in haskell. In this reguard its quite good -- it shows you step by step how some programming problems are broken down and attacked in pieces. Its presented in a socratic way so that you mostly find the way along yourself with some gentle guidance. I'm most of the way through it right now and I've found it quite useful, despite the fact that most of the examples are simplistic and I already know the haskell subjects being discussed.

The Economy of Phishing

From one of our own...

http://firstmonday.org/issues/issue10_9/abad/

Exploiting the Hierarchical Structure for Link Analysis

The rapid growth of the web and the ever increasing size of search results (along with their pollution by link spammers) means that new search and present methods need to be deployed. Here's a paper from MSR.

Link analysis algorithms have been extensively used in Web information retrieval. However, current link analysis algorithms generally work on a flat link graph, ignoring the hierarchal structure of the Web graph. They often suffer from two problems: the sparsity of link graph and biased ranking of newly-emerging pages. In this paper, we propose a novel ranking algorithm called Hierarchical Rank as a solution to these two problems, which considers both the hierarchical structure and the link structure of the Web. In this algorithm, Web pages are first aggregated based on their hierarchical structure at directory, host or domain level and link analysis is performed on the aggregated graph. Then, the importance of each node on the aggregated graph is distributed to individual pages belong to the node based on the hierarchical structure. This algorithm allows the importance of linked Web pages to be distributed in the Web page space even when the space is sparse and contains new pages. Experimental results on the .GOV collection of TREC 2003 and 2004 show that hierarchical ranking algorithm consistently outperforms other well-known ranking algorithms, including the PageRank, BlockRank and LayerRank. In addition, experimental results show that link aggregation at the host level is much better than link aggregation at either the domain or directory levels.

Source: Exploiting the Hierarchical Structure for Link Analysis, Gui-Rong Xue; Qiang Yang; Hua-Jun Zeng; Yong Yu; Zheng Chen.

Inductive Graph representation

Most people represent graphs with adjacency lists or as node objects with edges represented as references to other nodes. In this paper they represent a graph inductively. A graph is either an empty graph, or a graph with the addition of a new node and prev/next edges to existing nodes from the new node. This makes representation of some graph algorithms very elegant if you have the right primitives (ie. you need the ability to pull nodes out of the representation in an order that makes sense for your algorithm).

http://web.engr.oregonstate.edu/~erwig/papers/InductiveGraphs_JFP01.pdf

Several for September

I've been unable to post for a while so I have several diverse recommendations:

Some plan9 advocacy with good analysis of what happened to unix:
http://www.cs.unm.edu/~fastos/05meeting/PLAN9NOTDEADYET.pdf
http://herpolhode.com/rob/ugly.pdf
And a more detailed paper describing examples of how Plan9 leverages file servers to build a distributed system:
http://www.9con.org/rml/servers.pdf

An interesting talk about using recurrences and lazy evaluation to compute power series, and the paper it was based on:
http://herpolhode.com/rob/lec3.pdf
http://citeseer.ist.psu.edu/mcilroy89squinting.html
I goofed off with this concept a little in python to see how well generators work for this:
http://lava.net/~newsham/x/machine/powerseries.py
http://lava.net/~newsham/x/machine/powerseries2.py
And a tour de force on the same topic using haskell (which works much better for this purpose than python does). This paper is incredible:
http://citeseer.ist.psu.edu/mcilroy00music.html

While investigating Haskell (www.haskell.org) I came across this oldy but goody that tries to explain why anyone would want to use a functional language anyway. Seems like a good topic because I haven't heard many good reasons in the past.
http://www.md.chalmers.se/~rjmh/Papers/whyfp.pdf

Enjoy

Polygraph: Automatically Generating Signatures for Polymorphic Worms

Directed towards network based IDS (it seems.. have only skimmed), but fairly interesting. Found on wormblog :-)

Polygraph: Automatically Generating Signatures for Polymorphic Worms
http://www.ece.cmu.edu/~dawnsong/papers/polygraph.pdf

On a side note, anyone know if Dawn Song is the same Dawn Song that did the SSH session analysis research a few years back?

w00w00 Campathon Aug 19-21 Complete Details

HELLO THIS IS CHRISTOPHER ABADHERE IS THE OFFICIAL W00W00 CAMPATHON 2005 INVITATION

Instead of reading what is said below, you can also just go to the website, otherwise read on or simply erase.

http://the-mathclub.net/index.php/W00w00_campathon_2005

I know I missed a bunch of ppl on this mailing so anyone else you think who may be interested in attending this, goahead and spam their ass.

=====W00w00 campathon 2005=====

Alright bastards. Get ready for the w00w00 campathon 2005 in the StanislausNational Forest. This is going to be an event consisting of tents, fire,alcohol and much etc. The use of computers is most likely going to be gay.The ratio of men to women must remain optimal as to harbor an atmosphere ofcopius entertainment. A big sausage fest of nerd admins is the gayest thingthat I do not want to be at.

=====All the Ws=====

Who: All ElitesWhere: Stanislaus National Forest (http://www.fs.fed.us/r5/stanislaus/)When: August 19-21, 2005Campsite Directions: Will be emailed upon RSVP acceptance

=====Travel Arrangements=====

For anyone requiring rides to this thing, we will be arranging sorts of adhoc transportation from San Francisco on a first come first serve basis.Anyone else willing to drive can extend this ability to rideshare to otherswho cannot afford sweet sweet cars of their own. Anyone coming along fromoutside of the area is welcome to stay somewhere in San Francisco withsomeone who is willing to put up with your lameness for one night before thedeparture to the forest area. Additionally, if you are too cheap to afford ataxi from the airport to wherever you might be freeloading for one night ona couch before the voyage to outdoor-land, someone can arrange to pick yourlazy ass up.

=====Things You Need=====

You may in fact need a tentYou may in fact need a flash light and toilet paper to wipe you ass afteryou crap in a hole you did with a stick in the middle of the nightYou may need to bring food, as I will be unwilling to let you have any of mypepperoni sticks.You may need to bring a female companion, as magpie does not function wellas one.Beyond these basic things, I expect you to use your best judgement and bringawesome things, any additional magical ideas can be forwarded and thendeemed to be too stupid to list or maybe awesome enough to then be mentionedto everyone.

=====Contact and RSVP=====

If you have decided that youre not 100% lame and can handle doing somethingfucking awesome,
it will be in your best interest to rsvp us atcampathon@the-mathclub.net. There is absolutely no cost to this event,almost everyone is welcome, at the descrection of the secret biased andheavily judgemental committee. Feel free to let others know about thisevent, and have them attempt to RSVP to get involved in the coolness. If youare RSVPing as a group please just let us know all those involed, and howyou plan on getting there and if you need any travel arrangement assistance.RSVP to campathon@the-mathclub.net
Any other questions, you can just contact me
AIM: Ambient Empireemail: aempirei@the-mathclub.net

Finding Collisions in the Full SHA-1

Finding Collisions in the Full SHA-1
Xiaoyun Wang, Yiqun Lisa Yin, and Hongbo Yu

Discovered on: http://www.schneier.com/blog/

IBM's Billy Goat worm detection system

Lessons Learned from Billy Goat, an Accurate Worm-Detection System

This paper describes some of the lessons, insights and constructions stemming from the creation, deployment, and operation of the Billy Goat worm detection system. The most important feature of Billy Goat is its reliability in terms of accuracy, resilience and rapidity in detection and identification of worms without false positives. It is widely deployed throughout IBM and several other corporate networks. We discuss the features and requirements of worm detection systems in general, and how they are addressed by Billy Goat. We also describe some experiences and findings from our deployments of the system.

By: James Riordan; Diego Zamboni; Yann Duponchel

Published in: RZ3609 in 2005

Data clustering: a review

i'm just making up for a lack of posts here ... another good paper to read if you do any large scale data analysis and pattern recognition. i like this paper because it's a meaty, lengthy review and anyone can learn a boatload of useful material from it. highly recommended reading.

This paper presents an overview of pattern clustering methods from a statistical pattern recognition perspective, with a goal of providing useful advice and references to fundamental concepts accessible to the broad community of clustering practitioners. We present a taxonomy of clustering techniques, and identify cross-cutting themes and recent advances. We also describe some important applications of clustering algorithms such as image segmentation, object recognition, and information...

Source: Data clustering: a review, ACM Computing Surveys, Vol. 31, No. 3. (1999), pp. 264-323. Available for free.

CLUSTAL W: improving the sensitivity of progressive multiple sequence alignment ...

similar to the BLAST paper i posted some time ago, CLUSTAL W is a valuable tool and a great way to think about string matching.

The sensitivity of the commonly used progressive multiple sequence alignment method has been greatly improved for the alignment of divergent protein sequences. Firstly, individual weights are assigned to each sequence in a partial alignment in order to down-weight near-duplicate sequences and up-weight the most divergent ones. Secondly, amino acid substitution matrices are varied at different alignment stages according to the divergence of the sequences to be aligned. Thirdly, residue-specific gap penalties and locally reduced gap penalties in hydrophilic regions encourage new gaps in potential loop regions rather than regular secondary structure. Fourthly, positions in early alignments where gaps have been opened receive locally reduced gap penalties to encourage the opening up of new gaps at these positions. These modifications are incorporated into a new program, CLUSTAL W which is freely available.

Source: CLUSTAL W: improving the sensitivity of progressive multiple sequence alignment through sequence weighting, position-specific gap penalties and weight matrix choice. originallt published in Nucleic Acids Res, Vol. 22, No. 22. (11 November 1994), pp. 4673-4680. you can get it for free using the link above.

while this works well for things with obvious, formal structures and functions, it is difficult to achieve for arbitrary text and letters. still, i think that once you get beyond the definite match/no match mentality, regular expressions suddenly lose their appeal.

Windows Kernel Internals Process Architecture

"Windows Kernel Internals Process Architecture" PDF of PPT slides by David B. Probert, Ph.D; Windows Kernel Development, Microsoft Corporation.

http://www.i.u-tokyo.ac.jp/ss/lecture/new-documents/Lectures/13-Processes/Processes.pdf

This is specific to process & thread initialization and termination and goes into, while not ultra-specific, more of the gory details than most Windows internals books on this subject. In all honesty, I don't have the apropos character set installed, so I am not sure what version of Windows it is really meant to describe. I do, however, believe that while some ultra-specific data might change, much of the discussed is true today (with the exception of XP not calling static TLS initializers upon process startup -- IIRC, XP does not do that; it will call the initializers at thread creation/termination time and process termination time). It's pretty useful for RevEng'ing ntdll.dll.

The below link provides more PDF/PPT slides from this author on Windows internals and might be useful, but I have not really looked through them.

http://www.i.u-tokyo.ac.jp/ss/lecture/

Analyzing Memory Accesses in x86 Executables

i just had a look at a related paper to this one, but the abstract below is from a more general paper on the subject. if malware analysis interests you, this is probably worth a look.

This paper concerns static-analysis algorithms for analyzing x86 executables. The aim of the work is to recover intermediate representations that are similar to those that can be created for a program written in a high-level language. Our goal is to perform this task for programs such as plugins, mobile code, worms, and virus-infected code. For such programs, symbol-table and debugging information is either entirely absent, or cannot be relied upon if present; hence, the technique described in the paper makes no use of symboltable/debugging information. Instead, an analysis is carried out to recover information about the contents of memory locations and how they are manipulated by the executable.

Source: Analyzing Memory Accesses in x86 Executables, Gogul Balakrishnan and Thomas Reps.

w00w00 campathon

hello,we is planning a campathon (weekend camping trip) for the middle of august 2005in the stanislas national forest in california.http://www.fs.fed.us/r5/stanislaus/ pure eliteness.anyhow, anyone interested in such an event please let me know when you might possibly be available to do such a thing during the mid-august time frame.if you are not familiar with camping, pls refer to http://en.wikipedia.org/wiki/Camping

aempirei@the-mathclub.net

Minesweeper NP-complete (and more)

Yes, someone sat down and spent the time to figure out that some properties of playing minesweeper well are NP-complete:
http://web.mat.bham.ac.uk/R.W.Kaye/minesw/minesw.pdf

The reduction is pretty amusing, he shows that configurations of minesweeper act like logic gates and wires and gives a construction for building arbitrary logic systems out of minesweeper configurations, reducing the satisfiability of boolean expressions to solving problems in minesweeper.

There's also some more information in:
http://web.mat.bham.ac.uk/R.W.Kaye/minesw/ordmsw.htm
http://web.mat.bham.ac.uk/R.W.Kaye/minesw/ASE2003.pdf
http://web.mat.bham.ac.uk/R.W.Kaye/minesw/minesw.htm

The author goes further and describes how an infinite variation of minesweeper is Turing-complete in a paper linked from the last page. Kind of makes you evaluate what you're doing with your free time.

Another Google Paper

Here they're doing analysis on large sets of data distributed across many machines:
http://labs.google.com/papers/sawzall.html

The abstract describes it much better than I could, so I wont bother trying.

Two Papers

Here are two I ran across today. The first is a paper to be presented at usenix about protecting against exploits. I'm not sure how I feel about it. It seems rather hack-ish to me, but I havent been able to poke any major holes in the concept:
http://www.cs.arizona.edu/people/debray/papers/injection-attacks.ps

Two thoughts I had reguarding this paper are 1) it may be exponential to determine wether an instruction traps, but forcing it to trap is easy. It may be possible to use this fact to discover hidden system call traps. 2) They suggest modifying binaries to obfuscate them, however if they want to do this without losing the benefits of shared libraries, all binaries sharing the same library will have to share the same modifications. This means one binary may leak information about another. An attacker can use an information leak in one program to construct an exploit for a second program.

The second paper is about virtual machines and a technical called pre-virtualization. The idea is to use binary transformation of select parts of a system to get the effects of paravirtualization (ie. xen) without requiring modifications to the operating system source:
http://l4ka.org/projects/virtualization/afterburn/whitepaper.pdf

Metafor

I haven't gotten to the paper yet, but I just saw the demo .MOV and all I can say is "WOW." This is cool. Metafor is a toy system for programming in english:
http://web.media.mit.edu/~hugo/research/index.html#metafor

The site lists several papers and has a .MOV demo showing the system in use:
http://web.media.mit.edu/~hugo/demos/metafor-bartender-simple.mov

I am somewhat interested in this topic but as yet ignorant of the literature. If you know of some cool papers or background research in this area please post some info here or in the comments.

Cache Missing for Fun and Profit

I am sure most of you have seen this already, but in case not:

http://www.daemonology.net/papers/htt.pdf

Basically showing how HTT (on certain processors) can help increase chances of successful cache-timing attacks and uses OpenSSL's RSA implementation as an example on how to do the attack.

Collapsar: An attempt at effectively distributing honeypots

http://www.cs.purdue.edu/homes/dxu/pubs/Security04.pdf

Not too much new here; just a discussion of the current issues surrounding distributing honeypots and handling the data collection. They go on to talk about their methods of implementing a system they call Collapsar and give their results. Worthwhile if you're interested in what people are doing in this general field.

Papers on Lexical Cohesion and Semantic Analysis

Here's a few good papers, not new study, as lexical cohension as an idea in 1976 by hasan and halliday but good for thinking about analogies in reverse engineering and automated code analysis.

Jane Morris. "Lexical Cohesion Computed by Thesaural Relations as an Indicator of the Structure of Text." Computational Linguistics Volume 17, Number 1. 1991

http://acl.ldc.upenn.edu/J/J91/J91-1002.pdf

Hang Li. "Generalizing Case Frames Using a Thesaurus and the MDL Principle." Computational Linguistics Volume 24, Number 2. 1998.
http://acl.ldc.upenn.edu/J/J98/J98-2002.pdf

Viktor Pekar. "Modeling Semantic Coherence from Corpus Data: The Fact and the Frequency of a Co-occurrence." Coyote Papers 12, 1-8 Language in Cognitive Science. 1999.
http://coyotepapers.sbs.arizona.edu/Pekar.pdf

Amanda C. Jobbins, Lindsay J. Evett. "Text segmentation using reiteration and collocation." International Conference On Computational Linguistics. 1998.
http://acl.ldc.upenn.edu/P/P98/P98-1100.pdf

Using Rhythms of Relationships to Understand Email Archives

More on email and mining through boatloads of stored messages. I like this one because of the various levels of detail that get you down to a conversation.

Due to email’s ubiquitous nature, millions of users are intimate with the technology. However, most users are only familiar with managing their own email, which is an inherently different task than exploring an email archive. Historians and social scientists believe that email archives are important artifacts for understanding the individuals and communities they represent. In order to understand the conversations evidenced in an archive, context is needed. In this paper, we present a new way to gain this necessary context: analyzing the temporal rhythms of social relationships. We provide methods for constructing meaningful rhythms from the email headers by identifying relationships and interpreting their attributes. With these visualization techniques, email archive explorers can uncover insights that may have been otherwise hidden in the archive. We apply our methods to an individual’s fifteen-year email archive, which consists of about 45,000 messages and over 4,000 relationships.

Source: Using Rhythms of Relationships to Understand Email Archives, Adam Perer, Ben Shneiderman, Douglas W. Oard. Found via Smart Mobs. To appear at the Email Archive Visualization Workshop on June 2, 2005.

The "Help" user interface

It is sad that this paper is still relevant today. Written more than ten years ago, it describes the shortcomings of GUIs that we use today and offers an alternate approach:

http://www.fywss.com/plan9/plan9man/12help.ps.gz

The system described here is a precursor to the "acme" user interface that is currently shipped with plan9 and available on UN*X through
http://swtch.com/plan9port/.

Mapping weblog communities

It's been a while since I posted here, but this topic has been on my mind for a while. I'm attempting to discern interesting links within the InfosecDaily blog roll, aking to how Blogdex or Popdex does it. I have been reading a few papers on the subject and have a failing implementation at this point, but I need to go back to the drawing board. This paper caught my eye because it's a similar problem and problem-space. While it contains a few uninformed perceptions and conclusions about blog communities, overall it's just community discovery.

Websites of a particular class form increasingly complex networks, and new tools are needed to map and understand them. A way of visualizing this complex network is by mapping it. A map highlights which members of the community have similar interests, and reveals the underlying social network. In this paper, we will map a network of websites using Kohonen’s self-organizing map (SOM), a neural-net like method generally used for clustering and visualization of complex data sets. The set of websites considered has been the Blogalia weblog hosting site (based at http://www.blogalia.com/), a thriving community of around 200 members, created in January 2002. In this paper we show how SOM discovers interesting community features, its relation with other community-discovering algorithms, and the way it highlights the set of communities formed over the network.

Source: Mapping weblog communities, Juan J. Merelo-Guerv&oaccent;s, Beatriz Prieto. Fatima Rateb, and Fernando Tricas.

Content-rich biological network constructed by mining PubMed abstracts.

I suppose this might work for a variety of well formed abstracts. Someone could turn it loose on Citeseer and see what happens ... I've felt that the burgeoning quantity of data means that we have to work hard to jump across niches if we are to grow and prevent duplication of effort, to actually find progress, and keep ourselves inspired. Something like this might help.

The integration of the rapidly expanding corpus of information about the genome, transcriptome, and proteome, engendered by powerful technological advances, such as microarrays, and the availability of genomic sequence from multiple species, challenges the grasp and comprehension of the scientific community. Despite the existence of text-mining methods that identify biological relationships based on the textual co-occurrence of gene/protein terms or similarities in abstract texts, knowledge of the underlying molecular connections on a large scale, which is prerequisite to understanding novel biological processes, lags far behind the accumulation of data. While computationally efficient, the co-occurrence-based approaches fail to characterize (e.g., inhibition or stimulation, directionality) biological interactions. Programs with natural language processing (NLP) capability have been created to address these limitations, however, they are in general not readily accessible to the public. RESULTS: We present a NLP-based text-mining approach, Chilibot, which constructs content-rich relationship networks among biological concepts, genes, proteins, or drugs. Amongst its features, suggestions for new hypotheses can be generated. Lastly, we provide evidence that the connectivity of molecular networks extracted from the biological literature follows the power-law distribution, indicating scale-free topologies consistent with the results of previous experimental analyses. CONCLUSIONS: Chilibot distills scientific relationships from knowledge available throughout a wide range of biological domains and presents these in a content-rich graphical format, thus integrating general biomedical knowledge with the specialized knowledge and interests of the user. Chilibot http://www.chilibot.net can be accessed free of charge to academic users.

Source: Content-rich biological network constructed by mining PubMed abstracts., Chen H, Sharp BM. BMC Bioinformatics. 2004 Oct 08;5(1):147. You can also view the paper on BioMedCentral.

Lion's Commentary on UNIX

The infamous Lion's commentary is now available at


http://www.ercb.com/feature/feature.0067.html
or

http://www.lulu.com/content/99701

This book contains the entire source code of UNIX sixth edition with commentary.

If you're an OS buff, into retro computing and unix history or just trying to impress your friends, you should have this on your bookshelf.

Itanium - A System Implementor's Tale

https://www.disy.cse.unsw.edu.au/papers/disy/Gray_CCMH_05.pdf

This paper discusses some challenges faced by systems implementors when using the itanium. One of the more interesting sections discusses hand tuning the IPC mechanism in the L4 kernel. By hand-scheduling the code they were able to achieve better absolute (wall-time) and cycle performance than has been achieved on any other processor. A good read.

A concurrent window system

This one is an oldy but goody. Rob Pike describes his implementation
of a windowing system using newsqueak.

A concurrent window system.

The system uses a well defined interface between the windowing system
and the client that supports simple synchronous messages (rather than
asynchronous events typical of windowing systems). This is one of the many
windowing systems Pike has written and you can see the precursors of some
Plan9 and 8-1/2 ideas.

Practical File System Design

This is a book which I have not read, but it looks interesting
enough that I wish I had, and I feel comfortable putting it
on paperchasin where others will find it interesting.

Practical file system design with the Be File System by D. Giampaolo.

Ghostbuster from M$

Not sure if you guys saw this... It came across TH-research list and I hadn't seen it before.

File hiding is an advanced stealth technique that is becoming popular among system monitoring software such as RootKits, Trojans, and keyloggers. It presents a major challenge to system administrators and the anti-malware industry because detection and removal are virtually impossible if the target files are not even visible. In this paper, we present the Strider GhostBuster that exploits the fundamental weakness of the file-hiding behavior and turns the problem into its own solution. We have tested this diff-based tool successfully in the lab against several real-world system monitoring programs. The simplicity and effectiveness of the approach suggest that the following quote on the Internet may no longer be true: “When you can get the dir command to lie, it’s all over.” In the post-GhostBuster world: “The best way to hide is not trying to hide.”
Keywords: Rootkit, Trojan, keylogger, spyware, Gatekeeper, WinPE, stealth, system monitoring software

Cheers.

http://research.microsoft.com/research/pubs/view.aspx?type=Technical%20Report&id=775

Co-Validation: Using Model Disagreement to Validate Classification Algorithms

I've been keenly interested in measuring classification techniques, this one popped up on Yahoo! research:

In the context of binary classification, we define disagreement as a measure of how often two independently-trained models differ in their classification of unlabeled data. We show that per-instance disagreement is an unbiased estimate of the variance of error for that instance. We also show that disagreement provides a lower bound on the prediction (generalization) error, and a tight upper bound on the “variance of prediction error”, or the variance of the average error (across instances), where variance is measured across training sets. We explore the use of disagreement for error estimation and model selection. We call the procedure co-validation, since the two models effectively (in)validate one another by comparing results on unlabeled data, which we assume is relatively cheap and plentiful compared to labeled data. The procedure is especially effective in active learning settings, where training sets are not drawn at random and cross validation overestimates error. We present experimental results on several data sets exploring co-validation for active learning and model selection.

Source: Co-Validation: Using Model Disagreement to Validate Classification Algorithms, Omid Madani, David M. Pennock, GaryW. Flake.

Scatter/Gather: A Cluster-based Approach to Browsing Large Document Collections

More data mining techniques, these authors utilize an interesting, and evidently efficient, mechanism to discover related items in a data corpus. It's described as such:

As an alternative, the Scatter/Gather interface uses text clustering as a way to group document according to the overall similarities in their content. Scatter/Gather is so named because it allows the user to scatter documents into clusters, or groups, then gather a subset of these groups and re-scatter them to form new groups.

Each cluster in Scatter/Gather is represented by a list of topical terms, that is, a list of words that attempt to give the user the gist of what the documents in the cluster are about. The user can also look at the titles of the documents in each group. The documents can in the cluster can have other representations as well, such as summaries, or TileBars.

If a cluster still has too many documents, the user can re-cluster the documents in the cluster; that is, re-group that subset of documents into still smaller groups. This re-grouping process tends to change the kinds of themes of the clusters, because the documents in a subcollection discuss a different set of topics than all the documents in the larger collection.

You can read more on their technique on the scatter/gather website. Unlike so many neat ideas in literature, this one is openly implemented. Papers and examples.

CS == SocialScience

I was looking for some unrelated information on Jon Pincus' website when I came across this interesting position paper. It states that computer science these days is mostly social science. It also has lots of references to interesting examples of cross polination.

http://research.microsoft.com/users/jpincus/cs%20SocSci.html


Also check out some of his other papers. Good stuff.

Entanglement Teleportation Through 1D Heisenberg Chain

This is relayed in from aempirei; lazy bastard ;-)

Entanglement Teleportation Through 1D Heisenberg Chain

In the words of aE:
"it made perfect sense now where all my socks have gone"

FreeBSD: UFS/FFS snapshot from high up

UFS/FFS flow

Pretty neat, while hard to see unless you print it out... we've all seen this kind of thing before, but figured this one was worth posting.

Basic Local Alignment Search Tool

This is one of the core papers in bioinformatics (dating from 1990), but has implications beyond the life sciences. Recall an earlier post of mine on string distance metrics.

A new approach to rapid sequence comparison, basic local alignment search tool (BLAST), directly approximates alignments that optimize a measure of local similarity, the maximal segment pair (MSP) score. Recent mathematical results on the stochastic properties of MSP scores allow analysis of the performance of this method as well as the statistical significance of alignments it generates. The basic algorithm is simple and robust; it can be implemented in a number of ways and applied in a variety of contexts including straight-forward DNA and protein sequence database searches, motif searches, gene identification searches, and in the analysis of multiple regions of similarity in long DNA sequences. In addition to its flexibility and tractability to mathematical analysis, BLAST is an order of magnitude faster than existing sequence comparison tools of comparable sensitivity.

Source: Basic Local Alignment Search Tool, Altschul, S.F., Gish, W., Miller, W., Meyers, E.W., Lipman, D.J.

An Introduction to Bayesian Networks and their Contemporary Applications

While so many people equate Bayesian techniques with Bayesian classifiers, ie for spam filtering, it has signficantly more applications than just spam filtering. This is one of the seminal papers on the topic of Bayesian Networks.

Bayesian Networks are becoming an increasingly important area for research and application in the entire field of Artificial Intelligence. This paper explores the nature and implications for Bayesian Networks beginning with an overview and comparison of inferential statistics and Bayes' Theorem. The nature, relevance and applicability of Bayesian Network theory for issues of advanced computability forms the core of the current discussion. A number of current applications using Bayesian networks is examined. The paper concludes with a brief discussion of the appropriateness and limitations of Bayesian Networks for human-computer interaction and automated learning.

Source: An Introduction to Bayesian Networks and their Contemporary Applications, Daryle Niedermayer.

Haystack: A Platform for Creating, Organizing and Visualizing Information Using RDF

I'm actually not a big fan of the "semantic web" or much of the XML community. I find it's too deeply smothered in politics, posturing, and wasted time. However, sometimes useful things come out of even idle dreaming, and one of the things I like is how people are tackling information management. The Haystack project is an attempt to implement a functional RDF browser on top of the Eclipse platform. While I don't use it regularily, I do find I'm intrigued by the techniques in information management I see being attempted.

The Resource Definition Framework (RDF) is designed to support agent communication on the Web, but it is also suitable as a framework for modeling and storing personal information. Haystack is a personalized information repository that employs RDF in this manner. This flexible semistructured data model is appealing for several reasons. First, RDF supports ontologies created by the user and tailored to the user’s needs. At the same time, system ontologies can be specified and evolved to support a variety of high-level functionalities such as flexible organization schemes, semantic querying, and collaboration. In addition, we show that RDF can be used to engineer a component architecture that gives rise to a semantically rich and uniform user interface. We demonstrate that by aggregating various types of users’ data together in a homogeneous representation, we create opportunities for agents to make more informed deductions in automating tasks for users. Finally, we discuss the implementation of an RDF information store and a programming language specifically suited for manipulating RDF.

Source: Haystack: A Platform for Creating, Organizing and Visualizing Information Using RDF, David Huynh, David Karger, and Dennis Quan, Semantic Web Workshop 2002 Hawaii, USA.

Forecasting Uncertain Events with Small Groups

... predicting future outcomes that use small numbers of individuals participating in an imperfect information market...

Forecasting Uncertain Events with Small Groups - K-Y. Chen, L. R. Fine, B. A. Huberman

The Baldwin Effect in the Immune System: Learning by Somatic Hypermutation

I read this paper a few years ago when doing some research on new methods or approaches to developing HIDS (not sure I was really thinking about NIDS at the time). However, it is an interesting medical paper that might be applicable to other fields; at least some theory regarding "learned or acquired characteristics" that "could become part of the genetic makeup of succeeding generations..."

The Baldwin Effect in the Immune System: Learning by Somatic Hypermutation - R. Hightower, S. Forrest, A. S. Perelson

Might as well add this paper too:

Myths and Legends of the Baldwin Effect - Peter Turney of Institute for Information Technology,
National Research Council Canada

Model Selection and Multi-Model Inference

Model Selection and Multi-Model Inference

Whilst reading the String Distance Metrics paper that Jose posted, I had lots of thoughts going through head about some current work I'm doing. In my mind, part of all this relates to modelling data sets and things of that nature. I'm HORRIBLE at that... for now. The link above is not a paper, but a link to a book on Amazon. Essentially it goes through and talks about methods used to take inputs of a "good" scientific question and your data set and come up with a list of models that could be used to help you realize what you've got. It goes over Akaike's Information Criterion and things of that nature. I am definitely NOT a stats or modelling guy, so I figured this book would be of use to anyone who is starting to deal with large data sets and need to learn some things as to how to approach handling all that data.

Language Trees and Zipping

This paper presents a general approach to comparing two sets of information:
http://arxiv.org/abs/cond-mat/0108530.
They focus on strings of text, but the technique is quite powerful and general.

x86 Binary Reoptimization

http://www.crhc.uiuc.edu/IMPACT/ftp/report/impact-98-05.binary.pdf

"An Overview of the IMPACT x86 Binary Reoptimization Framework" by Matthew C. Merten and Michael S. Thiems. This, I think, is from 1998, so a bit old. Just found it interesting -- are there any tools like this for ELF format?

A Comparison of String Distance Metrics for Name-Matching Tasks

I've been looking at a lot of soft string matching metrics lately, this paper and the SecondString library have been interesting. A related project is MinorThird, "a collection of Java classes for storing text, annotating text, and learning to extract entities and categorize text." And finally, see Maxent, from a different group, described as a "Java package for training and using maximum entropy models." Too bad it's in Java ...

For more info on their methods and library, see their paper:

Using an open-source, Java toolkit of name-matching methods, we experimentally compare string distance metrics on the task of matching entity names. We investigate a number of different metrics proposed by different communities, including edit-distance metrics, fast heuristic string comparators , token-based distance metrics, and hybrid methods. Overall, the best-performing method is a hybrid scheme combining a TFIDF weighting scheme, which is widely used in information retrieval, with the Jaro-Winkler string-distance scheme, which was developed in the probabilistic record linkage community.

Source: A Comparison of String Distance Metrics for Name-Matching Tasks William W. Cohen, Pradeep Ravikumar, and Stephen E. Fienberg.

Permissive Action Links

http://www1.cs.columbia.edu/~smb/nsam-160/pal.html

A PAL -- a "Permissive Action Link" -- is the box that is supposed to prevent unauthorized use of a nuclear weapon. "Unauthorized" covers a wide range of sin, from terrorists who have stolen bombs to insane American military officers to our allies who may have some of their own uses for bombs that are covered by joint use agreements. It's supposed to be impossible to "hot-wire" a nuclear weapon. Is it?

Plan9 -- Reference documentation

High level plan9 docs.

http://whatexit.org/tal/mywritings/plan9_vs_telnet.pdf.

Newsham, North, and many others (Mike C. from xprime@#!#$) have been playing with p9 for awhile, so figured I'd put up links.

Win2k Loader "internals" paper

Been doing some research in this area so I figured I'd shoot it across. I haven't read it yet, so could be crap (sorry). But essentially discusses some of the DLL loading mechanisms.

What goes on inside Windows 2000: Solving the mysteries of the loader -- Russ Osterlund.

The Human Immune System and Network Intrusion Detection

This paper is pretty basic, but I figured I'd post just to throw some thoughts out there. A few years ago I had been really into reading medical papers on the human immune system... stepping back into that realm.

immune1.pdf

Not much..

Data Tastes Better Seasoned: Introducing the ASH Family of Hashing Algorithms

I'm no cryptographer, but this looks interesting.

Over the recent months it has become clear that the current generation of cryptographic hashing algorithms are insufficient to meet future needs. The ASH family of algorithms provides modifications to the existing SHA-2 family. These modifications are designed with two main goals: 1) Providing increased collision resistance. 2) Increasing mitigation of security risks post-collision. The unique public/private sections and salt/pepper design elements provide increased flexibility for a broad range of applications. The ASH family is a new generation of cryptographic hashing algorithms.

Source: Data Tastes Better Seasoned: Introducing the ASH Family of Hashing Algorithms, D.J. Capelis (via arXiv).

Silver Bullet

http://users.adelphia.net/~lilavois/Cosas/Reliability.htm.
Silver bullet to fix software reliability and scalability.

NCryptFS: A Secure and Convenient Cryptographic File System

Since I was reading the GBDE paper and I believe silitek was playing with NCryptFS, I figured I should read the paper. I've note yet read, so ... don't blame me! :-) J/k

Link to paper

Data Communications: The first 2500 years.

G. Holzmann
Data
Communications: The first 2500 years

Yes, its all old-hat.

GBDE - GEOM Based Disk Encryption

FreeBSD based, but was told by others that there was interest in porting to plan9. Since I had browsed this paper awhile ago but never read ... and there was porting interest, figured I'd re-read. Here's the link:


GBDE - Poul-Henning Kamp